Bug 2175230

Summary: SELinux AVC denials during podman run
Product: Red Hat Enterprise Linux 9 Reporter: Scott Wickersham <swickers>
Component: container-selinuxAssignee: Jindrich Novy <jnovy>
Status: ASSIGNED --- QA Contact: Edward Shen <weshen>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.0CC: ajia, dwalsh, jnovy, lsm5, lvrabec, mboddu, mmalik, rhartman, tsweeney, vmojzis, ypu, zpytela
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Scott Wickersham 2023-03-03 15:55:55 UTC
Description of problem: On RHEL-9.0 VM, Podman containers immediately exit with code (127) - 
SELinux is preventing /usr/bin/bash from read access on the file /usr/lib64/libc.so.6. 
SELinux is preventing /usr/bin/bash from getattr access on the chr_file /dev/pts/0.

Downgrading to container-selinux-2.188.0-1.el9_0.noarch and then running `restorecon -R -v $HOME` resolves the issue.

Version-Release number of selected component (if applicable):
podman-2:4.2.0-11.el9_1.x86_64
container-selinux-3:2.189.0-1.el9.noarch
crun-1.5-1.el9.x86_64

RHEL-9.0.0-x86_64-released

How reproducible:
Every time

Steps to Reproduce:
1. Fresh install of RHEL 9.0
2. Enable RHSM Subscription
3. `sudo dnf install podman`
4. `podman run -it ubi9 /bin/bash` (or any podman container run)

*If step 4 fails due to a `crun` issue, so you might have to downgrade crun between steps 3-4 to:
crun-1.4.5-2.el9_0.x86_64

Then it will fail with SELinux denials

Actual results:
The container immediately exits(127) instead of entering interactive shell

Expected results:
Interactive container /bin/bash prompt should show and not exit

Additional info:
Before getting the SELinux denial, I was getting the following error:
`Error: OCI runtime error: crun: /usr/bin/crun: symbol lookup error: /usr/bin/crun: undefined symbol: criu_feature_check`
Downgrading `crun` and `container-selinux` then running restorecon resolved my issues.




--------------

SELinux is preventing /usr/bin/bash from read access on the file /usr/lib64/libc.so.6.

*****  Plugin restorecon (99.5 confidence) suggests   ************************

If you want to fix the label.
/usr/lib64/libc.so.6 default label should be lib_t.
Then you can run restorecon. The access attempt may have been stopped due to insufficient permissions to access a parent directory in which case try to change the following command accordingly.
Do
# /sbin/restorecon -v /usr/lib64/libc.so.6

*****  Plugin catchall (1.49 confidence) suggests   **************************

If you believe that bash should be allowed read access on the libc.so.6 file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'bash' --raw | audit2allow -M my-bash
# semodule -X 300 -i my-bash.pp


Additional Information:
Source Context                system_u:system_r:container_t:s0:c296,c890
Target Context                unconfined_u:object_r:data_home_t:s0
Target Objects                /usr/lib64/libc.so.6 [ file ]
Source                        bash
Source Path                   /usr/bin/bash
Port                          <Unknown>
Host                          swickers-test-create-from-ansible-playbook
Source RPM Packages           bash-5.1.8-4.el9.x86_64
Target RPM Packages           glibc-2.34-28.el9_0.x86_64
SELinux Policy RPM            selinux-policy-targeted-34.1.29-1.el9_0.noarch
Local Policy RPM              selinux-policy-targeted-34.1.29-1.el9_0.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     swickers-test-create-from-ansible-playbook
Platform                      Linux swickers-test-create-from-ansible-playbook
                              5.14.0-70.13.1.el9_0.x86_64 #1 SMP PREEMPT Thu Apr
                              14 12:42:38 EDT 2022 x86_64 x86_64
Alert Count                   1
First Seen                    2023-03-02 12:49:08 EST
Last Seen                     2023-03-02 12:49:08 EST
Local ID                      919008f6-1fdf-43be-9544-ca2adcdf2412

Raw Audit Messages
type=AVC msg=audit(1677779348.848:1403): avc:  denied  { read } for  pid=22472 comm="bash" path="/usr/lib64/libc.so.6" dev="vda4" ino=14607 scontext=system_u:system_r:container_t:s0:c296,c890 tcontext=unconfined_u:object_r:data_home_t:s0 tclass=file permissive=0


type=SYSCALL msg=audit(1677779348.848:1403): arch=x86_64 syscall=mprotect success=no exit=EACCES a0=7f85fe9ae000 a1=1ce000 a2=0 a3=1 items=0 ppid=22470 pid=22472 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=(none) ses=3 comm=bash exe=/usr/bin/bash subj=system_u:system_r:container_t:s0:c296,c890 key=(null)

Hash: bash,container_t,data_home_t,file,read


------------------------------------------------- SECOND AVC AFTER RECTIFYING THE ONE ABOVE:  -----------------------------------------------------------------------------

SELinux is preventing /usr/bin/bash from getattr access on the chr_file /dev/pts/0.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that bash should be allowed getattr access on the 0 chr_file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'bash' --raw | audit2allow -M my-bash
# semodule -X 300 -i my-bash.pp


Additional Information:
Source Context                system_u:system_r:container_t:s0:c440,c767
Target Context                system_u:object_r:container_file_t:s0:c440,c767
Target Objects                /dev/pts/0 [ chr_file ]
Source                        bash
Source Path                   /usr/bin/bash
Port                          <Unknown>
Host                          swickers-test-create-from-ansible-playbook
Source RPM Packages           bash-5.1.8-4.el9.x86_64
Target RPM Packages
SELinux Policy RPM            selinux-policy-targeted-34.1.29-1.el9_0.noarch
Local Policy RPM              selinux-policy-targeted-34.1.29-1.el9_0.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     swickers-test-create-from-ansible-playbook
Platform                      Linux swickers-test-create-from-ansible-playbook
                              5.14.0-70.13.1.el9_0.x86_64 #1 SMP PREEMPT Thu Apr
                              14 12:42:38 EDT 2022 x86_64 x86_64
                              14 12:42:38 EDT 2022 x86_64 x86_64
Alert Count                   1
First Seen                    2023-03-02 13:13:24 EST
Last Seen                     2023-03-02 13:13:24 EST
Local ID                      130f9795-70f3-4c53-a077-10cd53d57d08

Raw Audit Messages
type=AVC msg=audit(1677780804.380:1489): avc:  denied  { getattr } for  pid=22713 comm="bash" path="/dev/pts/0" dev="devpts" ino=3 scontext=system_u:system_r:container_t:s0:c440,c767 tcontext=system_u:object_r:container_file_t:s0:c440,c767 tclass=chr_file permissive=0


type=SYSCALL msg=audit(1677780804.380:1489): arch=x86_64 syscall=newfstatat success=no exit=EACCES a0=0 a1=7f7848109f35 a2=7ffebc92abd0 a3=1000 items=0 ppid=22711 pid=22713 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=3 comm=bash exe=/usr/bin/bash subj=system_u:system_r:container_t:s0:c440,c767 key=(null)

Hash: bash,container_t,container_file_t,chr_file,getattr

Comment 1 rhartman 2023-03-03 16:43:05 UTC
Worth noting that this is occurring while trying to run podman rootless, with proper subuid/subgid setup

Comment 2 Scott Wickersham 2023-03-03 17:17:10 UTC
(In reply to rhartman from comment #1)
> Worth noting that this is occurring while trying to run podman rootless,
> with proper subuid/subgid setup

Yes.
And to add to that, I just tested running podman as root user and received the same `crun` error.  Downgrading `crun` resolved the issue for root user and there were no SELinux complaints.
However, I created a new user and added them to the wheel group and podman still failed for that user with the above AVC Denials. Downgrading container-selinux and running restorecon resolves the issue for this user.

Comment 3 Daniel Walsh 2023-03-03 18:49:47 UTC
Could you do 

yum reinstall container-selinux -y

And see if this blows up.

If this is successful could you run retstorecon -R -v $HOME/.local/share/containers

Comment 4 Scott Wickersham 2023-03-03 18:59:26 UTC
(In reply to Daniel Walsh from comment #3)
> Could you do 
> 
> yum reinstall container-selinux -y
> 
> And see if this blows up.
> 
> If this is successful could you run retstorecon -R -v
> $HOME/.local/share/containers

Reinstalled and ran the restorecon line you suggested and the same error persists.

The reinstall had some errors, but succeeded. See output below:

Running transaction
  Preparing        :                                                                                                                                                                 1/1 
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                                                                                                        1/2 
  Reinstalling     : container-selinux-3:2.189.0-1.el9.noarch                                                                                                                        1/2 
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                                                                                                        1/2 
libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
container: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20
container: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
container: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
/usr/sbin/semodule:  Failed!

  Cleanup          : container-selinux-3:2.189.0-1.el9.noarch                                                                                                                        2/2 
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                                                                                                        2/2 
  Verifying        : container-selinux-3:2.189.0-1.el9.noarch                                                                                                                        1/2 
  Verifying        : container-selinux-3:2.189.0-1.el9.noarch                                                                                                                        2/2 
Installed products updated.

Reinstalled:
  container-selinux-3:2.189.0-1.el9.noarch                                                                                                                                               

Complete!

Comment 5 Daniel Walsh 2023-03-07 13:41:27 UTC
The installation of container-selinux is blowing up.

libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
container: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20
container: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
container: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
/usr/sbin/semodule:  Failed!

This was caused by selinux-policy dropping support for certain socket types, I believe.  I also think that it is fixed in newer
version of selinux-policy and container-selinux.

Comment 6 Daniel Walsh 2023-03-07 13:42:05 UTC
Could you check to see if there is an updated selinux-policy available on RHEL9?

Comment 7 Scott Wickersham 2023-03-07 14:00:42 UTC
Looks like that was the issue! 

selinux-policy-34.1.29-1.el9_0.noarch upgraded to 34.1.43-1.el9_1.2.

Re-ran `yum reinstall container-selinux -y` without errors       (container-selinux-2.189.0-1.el9.noarch)

re-ran restorecon

and podman runs without error. 

Thank you!

Comment 8 Milos Malik 2023-03-07 14:18:50 UTC
# rpm -qa selinux\*
selinux-policy-38.1.8-1.el9.noarch
selinux-policy-targeted-38.1.8-1.el9.noarch
selinux-policy-mls-38.1.8-1.el9.noarch
selinux-policy-devel-38.1.8-1.el9.noarch
# rpm -qa | grep cont
# yum install container-selinux
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 1:42:52 ago on Tue 07 Mar 2023 07:26:45 AM EST.
Dependencies resolved.
================================================================================
 Package               Arch       Version              Repository          Size
================================================================================
Installing:
 container-selinux     noarch     3:2.199.0-1.el9      rhel-AppStream      55 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 55 k
Installed size: 62 k
Is this ok [y/N]: y
Downloading Packages:
container-selinux-2.199.0-1.el9.noarch.rpm      4.1 MB/s |  55 kB     00:00    
--------------------------------------------------------------------------------
Total                                           3.6 MB/s |  55 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Running scriptlet: container-selinux-3:2.199.0-1.el9.noarch               1/1 
  Installing       : container-selinux-3:2.199.0-1.el9.noarch               1/1 
  Running scriptlet: container-selinux-3:2.199.0-1.el9.noarch               1/1 
  Verifying        : container-selinux-3:2.199.0-1.el9.noarch               1/1 
Installed products updated.

Installed:
  container-selinux-3:2.199.0-1.el9.noarch                                      

Complete!
#

The problem is not reproducible on latest RHEL-9.2.

Comment 9 Zdenek Pytela 2023-03-07 16:10:04 UTC
Scott,

I am unable toreproduce any problem using different images.

Using 1MT-RHEL-9.0.0-released, packages version:
# rpm -qa "*selinux*" crun
libselinux-3.3-2.el9.x86_64
libselinux-utils-3.3-2.el9.x86_64
rpm-plugin-selinux-4.16.1.3-12.el9_0.x86_64
selinux-policy-34.1.29-1.el9_0.noarch
selinux-policy-targeted-34.1.29-1.el9_0.noarch
python3-libselinux-3.3-2.el9.x86_64
container-selinux-2.179.1-1.el9_0.noarch
crun-1.4.4-2.el9_0.x86_64
# ls /etc/selinux/targeted/policy/
policy.33
# podman run -it ubi9 /bin/bash
Resolved "ubi9" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi9:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 2a625e4afab5 done
Copying config 9877f06ecc done
Writing manifest to image destination
Storing signatures
# ls -Z
system_u:object_r:container_file_t:s0:c213,c641 afs
system_u:object_r:container_file_t:s0:c213,c641 bin
...

Do you happen to know which conditions are needed to trigger this issue?

Certainly some container versions cannot be used in RHEL 9.0 directly:
# dnf update container-selinux-2.198.0-1.el9.noarch.rpm
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:08:21 ago on Tue 07 Mar 2023 11:00:19 AM EST.
Error:
 Problem: cannot install the best update candidate for package container-selinux-3:2.179.1-1.el9_0.noarch
  - nothing provides selinux-policy >= 38.1.2-1.el9 needed by container-selinux-3:2.198.0-1.el9.noarch
  - nothing provides selinux-policy-base >= 38.1.2-1.el9 needed by container-selinux-3:2.198.0-1.el9.noarch
  - nothing provides selinux-policy-targeted >= 38.1.2-1.el9 needed by container-selinux-3:2.198.0-1.el9.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Comment 10 Zdenek Pytela 2023-03-07 16:49:03 UTC
Confirming the problem (#c0, #c4) after a quick session with Scott.

On a system which appears to be the same RHEL-9.0.0-released as I used, the dnf install podman command effects in installing
podman-4.2.0-11.el9_1
which requires
container-selinux-2.189.0-1.el9
During the update, the policydb error appears indicating the container module was created using newer version of libsepol.

As a quick workaround, I suggest using container-selinux-2.188.0-1.el9_0.noarch.rpm which installs well in RHEL 9.0.

My comment (#c9) works with a different version as I misread it (189 vs 198), sorry for that.

Comment 11 Zdenek Pytela 2023-03-08 09:32:25 UTC
Switching the component although I am not sure what exactly is the triggering condition. Possible solution is an rpm dependency, or to build the rpm not using the newest libsepol.

The problem is that when container-selinux-2.189.0-1.el9 is available for RHEL 9.0, it is not ruled out of installation, but the 
installation fails instead and the module is not updated, still the rpm transaction is reported successful. In the original description, the package was available for the dnf command, anyway this just should not happen.

# rpm -qa redhat-release selinux-policy container-selinux
redhat-release-9.0-2.17.el9.x86_64
selinux-policy-34.1.29-1.el9_0.noarch
container-selinux-2.179.1-1.el9_0.noarch
# wget https://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/container-selinux/2.189.0/1.el9/noarch/container-selinux-2.189.0-1.el9.noarch.rpm
# dnf update container-selinux-2.189.0-1.el9.noarch.rpm

Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:00:46 ago on Wed 08 Mar 2023 04:10:18 AM EST.
Dependencies resolved.
===================================================================================================
 Package                    Architecture    Version                    Repository             Size
===================================================================================================
Upgrading:
 container-selinux          noarch          3:2.189.0-1.el9            @commandline           47 k

Transaction Summary
===================================================================================================
Upgrade  1 Package

Total size: 47 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                           1/1
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                  1/2
  Upgrading        : container-selinux-3:2.189.0-1.el9.noarch                                  1/2
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                  1/2
libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
container: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20
container: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
container: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
/usr/sbin/semodule:  Failed!

  Cleanup          : container-selinux-3:2.179.1-1.el9_0.noarch                                2/2
  Running scriptlet: container-selinux-3:2.179.1-1.el9_0.noarch                                2/2
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                  2/2
  Verifying        : container-selinux-3:2.189.0-1.el9.noarch                                  1/2
  Verifying        : container-selinux-3:2.179.1-1.el9_0.noarch                                2/2
Installed products updated.

Upgraded:
  container-selinux-3:2.189.0-1.el9.noarch

Complete!
#

Comment 12 Daniel Walsh 2023-03-08 13:57:56 UTC
container-selinux is just using the libsepol that is in its buildroot.  If we should not be building with it, then it should not be in the build root.
If it is going to be updated, then we need to pull in an selinux-policy that is built with the new libsepol, correct?

Comment 13 Zdenek Pytela 2023-03-08 14:28:42 UTC
Vito,

Do you happen to know how this can happen (#c11)?
Is there a way we can use rpm dependency feature to prevent such a package from being installed?

Comment 14 Tom Sweeney 2023-03-08 19:41:12 UTC
Maybe Jindrich knows what needs to happen, but it's not totally clear to me.  Is there a version of libsepol that Jindrich should be looking at when doing the builds?

Comment 15 Vit Mojzis 2023-03-09 09:21:10 UTC
(In reply to Zdenek Pytela from comment #13)
> Vito,
> 
> Do you happen to know how this can happen (#c11)?
> Is there a way we can use rpm dependency feature to prevent such a package
> from being installed?

Policydb version was increased to 21 in userspace 3.4 (libsepol-3.4-1), released in rhel 9.1.

Based on [1], container-selinux-3:2.189.0-1.el9.noarch should not be available in rhel-9.0. And in my attempts I had to download it manually - dnf only installs container-selinux-3:2.188.0-1.el9_0.noarch, which does not have the problem.

I'm wondering if the new centos first [2] workflow could be to blame. It can make a single version of a package available in multiple releases (without rebuild). So container-selinux built in rhel-9.1 builtroot could possibly end up in rhel 9.0 if I understand it correctly (note podman-2:4.2.0-11.el9_1.x86_64 in the bug description). Though I don't see container-selinux-3:2.189.0-1.el9.noarch in any rhel-9.0 release and dnf doesn't want to install it on it's own.

As for rpm dependencies, we could add "Conflicts libsepol < libsepol-3.4" to container-selinux in rhel 9.1 an further, but that still leaves container-selinux-3:2.189.0-1.el9.noarch unchanged -- we need to figure out how it got to the rhel 9.0 machine in the first place.

[1] - https://pkgs.devel.redhat.com/cgit/rpms/container-selinux/log/?h=rhel-9.0.1
[2] https://docs.google.com/document/d/1n8URvrKtuZBT5K5Mwy9E6T5OVGwjwfeFO2P0ougccsA/edit#heading=h.rqcmps724dox

Comment 16 Jindrich Novy 2023-03-10 14:24:43 UTC
Adding "Conflicts: libsepol < 3.4" sounds good if it fixes the problem. I'll leave this to Tom to decide if backport is needed.

Comment 17 Zdenek Pytela 2023-03-10 14:57:58 UTC
(In reply to Jindrich Novy from comment #16)
> Adding "Conflicts: libsepol < 3.4" sounds good if it fixes the problem. I'll
Good, thank you. While there, can you also ensure macros are used wherever appropriate? 
https://fedoraproject.org/wiki/SELinux/IndependentPolicy#The_Preamble

The previously mentioned container-selinux-2.189.0-1.el9 contains
%global selinux_policyver 3.14.3-80.el8
BuildRequires: selinux-policy >= %{selinux_policyver}
which definitely does not feel right.

Thinking about it, following the guidelines perhaps would suffice?
In selinux-policy, we already have checks when new SELinux userspace is needed.

> leave this to Tom to decide if backport is needed.
Certainly; it'd be good to know how to reproduce the original report. I wasn't, and the troublesome package never appeared for updating in RHEL 9.0 using dnf.

Comment 18 Zdenek Pytela 2023-03-10 15:00:32 UTC
Scott,

Have you managed to find out when the particular container-selinux version is requested, or what is the triggering condition?

Comment 19 Scott Wickersham 2023-03-11 02:04:54 UTC
(In reply to Zdenek Pytela from comment #18)
> Scott,
> 
> Have you managed to find out when the particular container-selinux version
> is requested, or what is the triggering condition?


Hello Zdenek,
The output below is from a fresh RHEL 9.0 VM that I just spun up.
The only package install command I'm running is `dnf install podman` which is installing container_selinux 3:2.189.0-1.el9

-------------------------------------------------------------------

[cloud-user@rhel9-container-selinux-test ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.0 (Plow)

[cloud-user@rhel9-container-selinux-test ~]$ yum repolist
Failed to set locale, defaulting to C.UTF-8
Not root, Subscription Management repositories not updated
repo id                                                                          repo name
rhel-9-for-x86_64-appstream-rpms                                                 Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
rhel-9-for-x86_64-baseos-rpms                                                    Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)

[cloud-user@rhel9-container-selinux-test ~]$ sudo dnf install podman
Failed to set locale, defaulting to C.UTF-8
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                  25 MB/s |  17 MB     00:00    
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                                     15 MB/s | 8.6 MB     00:00    
Dependencies resolved.
=========================================================================================================================================================================================
 Package                                     Architecture                    Version                                     Repository                                                 Size
=========================================================================================================================================================================================
Installing:
 podman                                      x86_64                          2:4.2.0-11.el9_1                            rhel-9-for-x86_64-appstream-rpms                           12 M
Installing dependencies:
 conmon                                      x86_64                          2:2.1.4-1.el9                               rhel-9-for-x86_64-appstream-rpms                           55 k
 container-selinux                           noarch                          3:2.189.0-1.el9                             rhel-9-for-x86_64-appstream-rpms                           53 k
 containers-common                           x86_64                          2:1-49.el9_1                                rhel-9-for-x86_64-appstream-rpms                          121 k
 criu                                        x86_64                          3.15-13.el9                                 rhel-9-for-x86_64-appstream-rpms                          512 k
 criu-libs                                   x86_64                          3.15-13.el9                                 rhel-9-for-x86_64-appstream-rpms                           33 k
 crun                                        x86_64                          1.5-1.el9                                   rhel-9-for-x86_64-appstream-rpms                          191 k
 fuse-common                                 x86_64                          3.10.2-5.el9                                rhel-9-for-x86_64-baseos-rpms                             9.3 k
 fuse-overlayfs                              x86_64                          1.9-1.el9_0                                 rhel-9-for-x86_64-appstream-rpms                           74 k
 fuse3                                       x86_64                          3.10.2-5.el9                                rhel-9-for-x86_64-appstream-rpms                           58 k
 fuse3-libs                                  x86_64                          3.10.2-5.el9                                rhel-9-for-x86_64-appstream-rpms                           94 k
 iptables-nft                                x86_64                          1.8.7-28.el9                                rhel-9-for-x86_64-baseos-rpms                             208 k
 libnet                                      x86_64                          1.2-6.el9                                   rhel-9-for-x86_64-appstream-rpms                           61 k
 libnftnl                                    x86_64                          1.2.2-1.el9                                 rhel-9-for-x86_64-baseos-rpms                              85 k
 libslirp                                    x86_64                          4.4.0-7.el9                                 rhel-9-for-x86_64-appstream-rpms                           72 k
 netavark                                    x86_64                          2:1.1.0-7.el9_1                             rhel-9-for-x86_64-appstream-rpms                          2.1 M
 nftables                                    x86_64                          1:1.0.4-9.el9_1                             rhel-9-for-x86_64-baseos-rpms                             405 k
 podman-catatonit                            x86_64                          2:4.2.0-11.el9_1                            rhel-9-for-x86_64-appstream-rpms                          351 k
 shadow-utils-subid                          x86_64                          2:4.9-5.el9                                 rhel-9-for-x86_64-baseos-rpms                              90 k
 slirp4netns                                 x86_64                          1.2.0-2.el9_0                               rhel-9-for-x86_64-appstream-rpms                           49 k
 yajl                                        x86_64                          2.1.0-21.el9_0                              rhel-9-for-x86_64-appstream-rpms                           42 k
Installing weak dependencies:
 aardvark-dns                                x86_64                          2:1.1.0-5.el9_1                             rhel-9-for-x86_64-appstream-rpms                          998 k

Transaction Summary
=========================================================================================================================================================================================
Install  22 Packages

Total download size: 18 M
Installed size: 59 M
Is this ok [y/N]: y
Downloading Packages:
(1/22): fuse3-libs-3.10.2-5.el9.x86_64.rpm                                                                                                               300 kB/s |  94 kB     00:00    
(2/22): criu-libs-3.15-13.el9.x86_64.rpm                                                                                                                 103 kB/s |  33 kB     00:00    
(3/22): criu-3.15-13.el9.x86_64.rpm                                                                                                                      1.6 MB/s | 512 kB     00:00    
(4/22): fuse3-3.10.2-5.el9.x86_64.rpm                                                                                                                    111 kB/s |  58 kB     00:00    
(5/22): libslirp-4.4.0-7.el9.x86_64.rpm                                                                                                                  332 kB/s |  72 kB     00:00    
(6/22): libnet-1.2-6.el9.x86_64.rpm                                                                                                                       66 kB/s |  61 kB     00:00    
(7/22): fuse-overlayfs-1.9-1.el9_0.x86_64.rpm                                                                                                            540 kB/s |  74 kB     00:00    
(8/22): yajl-2.1.0-21.el9_0.x86_64.rpm                                                                                                                   260 kB/s |  42 kB     00:00    
(9/22): slirp4netns-1.2.0-2.el9_0.x86_64.rpm                                                                                                             350 kB/s |  49 kB     00:00    
(10/22): crun-1.5-1.el9.x86_64.rpm                                                                                                                       898 kB/s | 191 kB     00:00    
(11/22): container-selinux-2.189.0-1.el9.noarch.rpm                                                                                                      231 kB/s |  53 kB     00:00    
(12/22): conmon-2.1.4-1.el9.x86_64.rpm                                                                                                                   340 kB/s |  55 kB     00:00    
(13/22): containers-common-1-49.el9_1.x86_64.rpm                                                                                                         718 kB/s | 121 kB     00:00    
(14/22): aardvark-dns-1.1.0-5.el9_1.x86_64.rpm                                                                                                           5.1 MB/s | 998 kB     00:00    
(15/22): netavark-1.1.0-7.el9_1.x86_64.rpm                                                                                                               9.2 MB/s | 2.1 MB     00:00    
(16/22): podman-catatonit-4.2.0-11.el9_1.x86_64.rpm                                                                                                      2.2 MB/s | 351 kB     00:00    
(17/22): iptables-nft-1.8.7-28.el9.x86_64.rpm                                                                                                            1.0 MB/s | 208 kB     00:00    
(18/22): fuse-common-3.10.2-5.el9.x86_64.rpm                                                                                                              30 kB/s | 9.3 kB     00:00    
(19/22): podman-4.2.0-11.el9_1.x86_64.rpm                                                                                                                 26 MB/s |  12 MB     00:00    
(20/22): shadow-utils-subid-4.9-5.el9.x86_64.rpm                                                                                                         698 kB/s |  90 kB     00:00    
(21/22): libnftnl-1.2.2-1.el9.x86_64.rpm                                                                                                                 348 kB/s |  85 kB     00:00    
(22/22): nftables-1.0.4-9.el9_1.x86_64.rpm                                                                                                               2.3 MB/s | 405 kB     00:00    
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                    8.6 MB/s |  18 MB     00:02     
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                                 3.5 MB/s | 3.6 kB     00:00    
Importing GPG key 0xFD431D51:
 Userid     : "Red Hat, Inc. (release key 2) <security>"
 Fingerprint: 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Importing GPG key 0x5A6340B3:
 Userid     : "Red Hat, Inc. (auxiliary key 3) <security>"
 Fingerprint: 7E46 2425 8C40 6535 D56D 6F13 5054 E4A4 5A63 40B3
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                 1/1 
  Installing       : libnftnl-1.2.2-1.el9.x86_64                                                                                                                                    1/22 
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                                                                                                       2/22 
  Installing       : container-selinux-3:2.189.0-1.el9.noarch                                                                                                                       2/22 
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                                                                                                       2/22 
libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
container: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20
container: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
container: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
/usr/sbin/semodule:  Failed!

  Installing       : fuse3-libs-3.10.2-5.el9.x86_64                                                                                                                                 3/22 
  Installing       : iptables-nft-1.8.7-28.el9.x86_64                                                                                                                               4/22 
  Running scriptlet: iptables-nft-1.8.7-28.el9.x86_64                                                                                                                               4/22 
  Installing       : nftables-1:1.0.4-9.el9_1.x86_64                                                                                                                                5/22 
  Running scriptlet: nftables-1:1.0.4-9.el9_1.x86_64                                                                                                                                5/22 
  Installing       : shadow-utils-subid-2:4.9-5.el9.x86_64                                                                                                                          6/22 
  Installing       : fuse-common-3.10.2-5.el9.x86_64                                                                                                                                7/22 
  Installing       : fuse3-3.10.2-5.el9.x86_64                                                                                                                                      8/22 
  Installing       : fuse-overlayfs-1.9-1.el9_0.x86_64                                                                                                                              9/22 
  Running scriptlet: fuse-overlayfs-1.9-1.el9_0.x86_64                                                                                                                              9/22 
  Installing       : aardvark-dns-2:1.1.0-5.el9_1.x86_64                                                                                                                           10/22 
  Installing       : netavark-2:1.1.0-7.el9_1.x86_64                                                                                                                               11/22 
  Installing       : conmon-2:2.1.4-1.el9.x86_64                                                                                                                                   12/22 
  Installing       : yajl-2.1.0-21.el9_0.x86_64                                                                                                                                    13/22 
  Installing       : libslirp-4.4.0-7.el9.x86_64                                                                                                                                   14/22 
  Installing       : slirp4netns-1.2.0-2.el9_0.x86_64                                                                                                                              15/22 
  Installing       : libnet-1.2-6.el9.x86_64                                                                                                                                       16/22 
  Installing       : criu-3.15-13.el9.x86_64                                                                                                                                       17/22 
  Installing       : criu-libs-3.15-13.el9.x86_64                                                                                                                                  18/22 
  Installing       : crun-1.5-1.el9.x86_64                                                                                                                                         19/22 
  Installing       : containers-common-2:1-49.el9_1.x86_64                                                                                                                         20/22 
  Installing       : podman-catatonit-2:4.2.0-11.el9_1.x86_64                                                                                                                      21/22 
  Installing       : podman-2:4.2.0-11.el9_1.x86_64                                                                                                                                22/22 
  Running scriptlet: container-selinux-3:2.189.0-1.el9.noarch                                                                                                                      22/22 
  Running scriptlet: podman-2:4.2.0-11.el9_1.x86_64                                                                                                                                22/22 
  Verifying        : fuse3-libs-3.10.2-5.el9.x86_64                                                                                                                                 1/22 
  Verifying        : libnet-1.2-6.el9.x86_64                                                                                                                                        2/22 
  Verifying        : criu-libs-3.15-13.el9.x86_64                                                                                                                                   3/22 
  Verifying        : fuse3-3.10.2-5.el9.x86_64                                                                                                                                      4/22 
  Verifying        : criu-3.15-13.el9.x86_64                                                                                                                                        5/22 
  Verifying        : libslirp-4.4.0-7.el9.x86_64                                                                                                                                    6/22 
  Verifying        : yajl-2.1.0-21.el9_0.x86_64                                                                                                                                     7/22 
  Verifying        : fuse-overlayfs-1.9-1.el9_0.x86_64                                                                                                                              8/22 
  Verifying        : slirp4netns-1.2.0-2.el9_0.x86_64                                                                                                                               9/22 
  Verifying        : container-selinux-3:2.189.0-1.el9.noarch                                                                                                                      10/22 
  Verifying        : crun-1.5-1.el9.x86_64                                                                                                                                         11/22 
  Verifying        : conmon-2:2.1.4-1.el9.x86_64                                                                                                                                   12/22 
  Verifying        : aardvark-dns-2:1.1.0-5.el9_1.x86_64                                                                                                                           13/22 
  Verifying        : netavark-2:1.1.0-7.el9_1.x86_64                                                                                                                               14/22 
  Verifying        : containers-common-2:1-49.el9_1.x86_64                                                                                                                         15/22 
  Verifying        : podman-catatonit-2:4.2.0-11.el9_1.x86_64                                                                                                                      16/22 
  Verifying        : podman-2:4.2.0-11.el9_1.x86_64                                                                                                                                17/22 
  Verifying        : fuse-common-3.10.2-5.el9.x86_64                                                                                                                               18/22 
  Verifying        : iptables-nft-1.8.7-28.el9.x86_64                                                                                                                              19/22 
  Verifying        : shadow-utils-subid-2:4.9-5.el9.x86_64                                                                                                                         20/22 
  Verifying        : libnftnl-1.2.2-1.el9.x86_64                                                                                                                                   21/22 
  Verifying        : nftables-1:1.0.4-9.el9_1.x86_64                                                                                                                               22/22 
Installed products updated.

Installed:
  aardvark-dns-2:1.1.0-5.el9_1.x86_64          conmon-2:2.1.4-1.el9.x86_64             container-selinux-3:2.189.0-1.el9.noarch          containers-common-2:1-49.el9_1.x86_64         
  criu-3.15-13.el9.x86_64                      criu-libs-3.15-13.el9.x86_64            crun-1.5-1.el9.x86_64                             fuse-common-3.10.2-5.el9.x86_64               
  fuse-overlayfs-1.9-1.el9_0.x86_64            fuse3-3.10.2-5.el9.x86_64               fuse3-libs-3.10.2-5.el9.x86_64                    iptables-nft-1.8.7-28.el9.x86_64              
  libnet-1.2-6.el9.x86_64                      libnftnl-1.2.2-1.el9.x86_64             libslirp-4.4.0-7.el9.x86_64                       netavark-2:1.1.0-7.el9_1.x86_64               
  nftables-1:1.0.4-9.el9_1.x86_64              podman-2:4.2.0-11.el9_1.x86_64          podman-catatonit-2:4.2.0-11.el9_1.x86_64          shadow-utils-subid-2:4.9-5.el9.x86_64         
  slirp4netns-1.2.0-2.el9_0.x86_64             yajl-2.1.0-21.el9_0.x86_64             

Complete!
[cloud-user@rhel9-container-selinux-test ~]$

Comment 20 Jindrich Novy 2023-03-13 08:45:32 UTC
Let me do a quick summary of this bug - the situation happens only with libsepol lesser than 3.4 which is present in RHEL9.0.0 only?

package   branch          version   upstream_branch  commit
---       ---             ---       ---              ---
libsepol  rhel-9.0.0      3.3-2     ---              ---
libsepol  rhel-9.1.0      3.4-1.1   ---              ---
libsepol  rhel-9-main     3.5-1     ---              ---

So to fix this zstream update to 9.0.0 would be required?

Comment 21 Vit Mojzis 2023-03-13 16:21:22 UTC
Yes, libsepol < 3.4 is only present in rhel 9.0.

Please note that the whole SELinux userspace (and probably also selinux-policy) would need to be updated and I'm not sure if that is the only solution here.
Maybe we could make sure container-selinux-3:2.189.0-1.el9 is not available there (since it shouldn't be based on [1]). container-selinux-2.188.0-1.el9 should be the latest for rhel-9.0 and more importantly, rhel-9.0 should not be getting packages built in rhel-9.1 buildroot.

The VM Scott is using appears to be getting packages from rhel-9.1 repositories. Note aardvark-dns-2:1.1.0-5.el9_1.x86_64, nftables-1:1.0.4-9.el9_1.x86_64 and podman-2:4.2.0-11.el9_1.x86_64. IMO fixing this inconsistency should be the main concern.

[1] - https://pkgs.devel.redhat.com/cgit/rpms/container-selinux/log/?h=rhel-9.0.1