Additional note: also reproducible on Fedora 36, with the following packages:
# rpm -qa 389-ds-base selinux-policy
389-ds-base-2.0.11-1.fc36.x86_64
selinux-policy-35.7-1.fc36.noarch

Florence,

We need to label /dev/shm/slapd-IPA-TEST and have a transition. Do you know which service creates this directory?

Hi Zdenek,

the directory /dev/shm/slapd-IPA-TEST is created when ipa-restore is run, during the step that calls
/usr/sbin/dsctl IPA-TEST ldif2db userRoot /var/lib/dirsrv/slapd-IPA-TEST/ldif/IPA-TEST-userRoot.ldif

If you need more details re. the internals of dsctl command, maybe @tbordaz can help

> F35 389-ds-base-2.0.11-1.fc35.x86_64
> (https://koji.fedoraproject.org/koji/buildinfo?buildID=1858720) contains the
> fix that moves the DB home under /dev/shm
> (https://github.com/389ds/389-ds-base/issues/2790).
> 
> The fix sets db_home_dir=/dev/shm/slapd-instance in
> /usr/share/dirsrv/inf/defaults.inf. This properties file is used by the
> command 'dscreate' when creating the instance.
> My understanding is that selinux in F35 requires similar fix that was done
> in selinux-policy-34.1.19-1.el9.
> 

Neither dscreate or dsctl commands are SELinux confined:
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0 3647 Dec 16 16:30 /usr/sbin/dscreate
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0 5604 Dec 16 16:30 /usr/sbin/dsctl

and the directory name is unpredictable:

f35# ls -lZa /dev/shm
total 8
drwxrwxrwt.  4 root   root   system_u:object_r:tmpfs_t:s0             120 Jan  5 14:11 .
drwxr-xr-x. 20 root   root   system_u:object_r:device_t:s0           4040 Jan  5 10:27 ..
-rw-------.  1 dirsrv dirsrv system_u:object_r:dirsrv_tmpfs_t:s0       32 Jan  5 14:10 sem.slapd-IPA-TEST.stats
drwx------.  2 dirsrv dirsrv unconfined_u:object_r:user_tmp_t:s0       60 Jan  5 14:10 slapd-IPA-TEST

so labeling of the directory cannot be handled in selinux-policy. The other example though:

> I use to run the following command to create an instance: 
> dscreate create-template | sed -e 's/;suffix =/suffix = dc=example,dc=com/'
> -e 's/;sample_entries = no/sample_entries = yes/' -e 's/;create_suffix_entry
> = False/create_suffix_entry = True/' -e 's/;root_password =
> Directory_Manager_Password/root_password = password/' >/tmp/foo2; dscreate
> from-file /tmp/foo2

seems to work:

drwxrwxrwt.  4 root   root   system_u:object_r:tmpfs_t:s0             120 Jan  5 14:11 .
drwxr-xr-x. 20 root   root   system_u:object_r:device_t:s0           4040 Jan  5 10:27 ..
-rw-------.  1 dirsrv dirsrv system_u:object_r:dirsrv_tmpfs_t:s0       32 Jan  5 14:11 sem.slapd-localhost.stats
drwxrwx---.  2 dirsrv dirsrv unconfined_u:object_r:dirsrv_tmpfs_t:s0  120 Jan  5 14:11 slapd-localhost

supposedly because it reports some relabeling:

Starting installation ...
Validate installation settings ...
Create file system structures ...
Create self-signed certificate database ...
Perform SELinux labeling ...
Create database backend: dc=example,dc=com ...
Perform post-installation tasks ...
Completed installation for instance: slapd-localhost

Is it possible to run similar relabeling in dscreate, or use other means to fix SELinux labels on /dev/shm/slapd-NAME?


f35# rpm -qa 389-ds-base freeipa-server selinux-policy
selinux-policy-35.7-1.fc35.noarch
389-ds-base-2.0.12-1.fc35.x86_64
freeipa-server-4.9.8-1.fc35.x86_64

if the info file indicates that selinux is enabled (/usr/share/dirsrv/inf/defaults.inf:with_selinux), dscreate label (via selinux.restorecon(path, recursive=True)) a set of paths including '/dev/shm' (aka db_home_directory) (see https://github.com/389ds/389-ds-base/blob/master/src/lib389/lib389/instance/setup.py#L911).
Although we can relabel it in some commands like dsctl I think it would be a kind of workaround.

if ipa-server-install labeled /dev/shm, then ipa-server-install -uninstall removed /dev/shm/slapd-IPA-TEST. Upon restore /dev/shm/slapd-IPA-TEST is recreated but is missing the label. Is that the scenario you are thinking of ?

(In reply to thierry bordaz from comment #6)
> if the info file indicates that selinux is enabled
> (/usr/share/dirsrv/inf/defaults.inf:with_selinux), dscreate label (via
> selinux.restorecon(path, recursive=True)) a set of paths including
> '/dev/shm' (aka db_home_directory) (see
> https://github.com/389ds/389-ds-base/blob/master/src/lib389/lib389/instance/
> setup.py#L911).
> Although we can relabel it in some commands like dsctl I think it would be a
> kind of workaround.
> 
> if ipa-server-install labeled /dev/shm, then ipa-server-install -uninstall
> removed /dev/shm/slapd-IPA-TEST. Upon restore /dev/shm/slapd-IPA-TEST is
> recreated but is missing the label. Is that the scenario you are thinking of
> ?

If /dev/shm/.* is created by a service which runs a confined executable, it can be and actually is handled in selinux-policy.
For a cli command we cannot do it, so I can think of 2 solutions - run restorecon in ipa-restore or have some parts of the task, which restore /dev/shm, as a systemd service, too.

Moving to 389-ds-base component as agreed with tbordaz

This was supposed to be fixed in this selinux-policy bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1843238

With this change this feature was working correctly so I'm surprised to see if popping up again.

Does this need to backported to Fedora zpytela?

(In reply to mreynolds from comment #9)
> This was supposed to be fixed in this selinux-policy bug:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1843238
> 
> With this change this feature was working correctly so I'm surprised to see
> if popping up again.
> 
> Does this need to backported to Fedora zpytela?

Sorry was a bit hasty in my reply, this is only happening after a IPA backup/restore (not standard server install).  Investigating, but I will need to know more about IPA's backup/restore process...

Hi Mark,
the code of ipa-backup is available here: https://pagure.io/freeipa/blob/master/f/ipaserver/install/ipa_backup.py
It mainly does commands equivalent to "dsctl <instance> db2ldif --replication <backend> ldiffile" (once for the domain backend, once for ipaca backend), then "dsctl <instance> db2bak backupdir".

The code of ipa-restore is available here: https://pagure.io/freeipa/blob/master/f/ipaserver/install/ipa_restore.py
It uses the files created by ipa-backup and calls "dsctl <instance> ldif2db <backend> ldiffile". This is the step that creates the /dev/shm/slapd-IPA-TEST.

Later ipa-restore calls "ipactl start" and the step starting 389-ds fails because it can't read /dev/shm/slapd-IPA-TEST/DBVERSION.

(In reply to Florence Blanc-Renaud from comment #11)
> Hi Mark,
> the code of ipa-backup is available here:
> https://pagure.io/freeipa/blob/master/f/ipaserver/install/ipa_backup.py
> It mainly does commands equivalent to "dsctl <instance> db2ldif
> --replication <backend> ldiffile" (once for the domain backend, once for
> ipaca backend), then "dsctl <instance> db2bak backupdir".
> 
> The code of ipa-restore is available here:
> https://pagure.io/freeipa/blob/master/f/ipaserver/install/ipa_restore.py
> It uses the files created by ipa-backup and calls "dsctl <instance> ldif2db
> <backend> ldiffile". This is the step that creates the
> /dev/shm/slapd-IPA-TEST.

Is IPA always calling "dsctl INSTANCE bak2db"?  If it is then we can fix it in our script (dsctl).  However, if IPA is manually doing the restore (meaning IPA is copying the files over itself) then DS can NOT fix this and IPA will need to do the restorecon itself.  Looking at the IPA source it does look like IPA is calling bak2db, but I don't know if it always calls "dsctl bak2db".  I will assume it is, but if there are cases where it is manually done by IPA then we will have problems.

(In reply to mreynolds from comment #10)
> (In reply to mreynolds from comment #9)
> > This was supposed to be fixed in this selinux-policy bug:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1843238
> > 
> > With this change this feature was working correctly so I'm surprised to see
> > if popping up again.
> > 
> > Does this need to backported to Fedora zpytela?
> 
> Sorry was a bit hasty in my reply, this is only happening after a IPA
> backup/restore (not standard server install).  Investigating, but I will
> need to know more about IPA's backup/restore process...

This time it is a different issue. Anyway almost always Fedora contains a superset of rules of RHEL.

Note so far we have managed to find 3 possible solutions:
- run restorecon after files restoring and before ipactl start
- include SELinux context or all extended attributes in both backup and restore
- handle restoring as a systemd service

(In reply to Florence Blanc-Renaud from comment #11)
> Hi Mark,
> the code of ipa-backup is available here:
> https://pagure.io/freeipa/blob/master/f/ipaserver/install/ipa_backup.py
> It mainly does commands equivalent to "dsctl <instance> db2ldif
> --replication <backend> ldiffile" (once for the domain backend, once for
> ipaca backend), then "dsctl <instance> db2bak backupdir".
> 
> The code of ipa-restore is available here:
> https://pagure.io/freeipa/blob/master/f/ipaserver/install/ipa_restore.py
> It uses the files created by ipa-backup and calls "dsctl <instance> ldif2db
> <backend> ldiffile". This is the step that creates the
> /dev/shm/slapd-IPA-TEST.
> 
> Later ipa-restore calls "ipactl start" and the step starting 389-ds fails
> because it can't read /dev/shm/slapd-IPA-TEST/DBVERSION.


I can not reproduce this with a standalone DS instance.

This is what I tried, did I miss a step?

# getenforce
Enforcing

# dsctl slapd-localhost stop
# dsctl slapd-localhost db2bak
-->  copy backup to /tmp

# dsctl slapd-localhost db2ldif userroot
--> copy LDIF to /tmp

# dsctl slapd-localhost remove --do-it
# dscreate from-file /data/dssetup.inf
# dsctl slapd-localhost stop
--> copy /tmp LDIF and bak to the proper locations under /var/lib/dirsrv/slapd-localhost/bak|ldif   Need to do this since the instance was removed and recreated

# dsctl slapd-localhost bak2db <path to bakup>
# dsctl slapd-localhost ldif2db userroot <path to ldif>
# dsctl slapd-localhost start
# ldapmodify ... ... succeeds

There are no errors and I can update database after starting it.  Did I do something wrong?

I reproduced this manually using the provided tests.

I'm not sure the root cause. I'm thinking that `dsctl ldif2db` doesn't set the contexts and perhaps dscreate does.

To test this theory I used ipa-restore to restore the data. The result was:

# ls -lRZ /dev/shm
/dev/shm:
total 0
drwx------. 2 dirsrv dirsrv unconfined_u:object_r:user_tmp_t:s0 120 Jan 13 16:49 slapd-EXAMPLE-TEST

/dev/shm/slapd-EXAMPLE-TEST:
total 220444
-rw-------. 1 dirsrv dirsrv system_u:object_r:user_tmp_t:s0      47874048 Jan 13 16:49 __db.001
-rw-------. 1 dirsrv dirsrv system_u:object_r:user_tmp_t:s0      16179200 Jan 13 16:49 __db.002
-rw-------. 1 dirsrv dirsrv system_u:object_r:user_tmp_t:s0     161677312 Jan 13 16:50 __db.003
-rw-------. 1 dirsrv dirsrv unconfined_u:object_r:user_tmp_t:s0        51 Jan 13 16:49 DBVERSION

This reproduces the failing test.

I used restorecon -R /dev/shm to fix the contexts. They were assigned as expected, so the policy I guess knows what things should be? Curious that they aren't created with the correct policy.

# ls -lRZ /dev/shm
/dev/shm:
total 0
drwx------. 2 dirsrv dirsrv unconfined_u:object_r:dirsrv_tmpfs_t:s0 120 Jan 13 16:49 slapd-EXAMPLE-TEST

/dev/shm/slapd-EXAMPLE-TEST:
total 220444
-rw-------. 1 dirsrv dirsrv system_u:object_r:dirsrv_tmpfs_t:s0      47874048 Jan 13 16:51 __db.001
-rw-------. 1 dirsrv dirsrv system_u:object_r:dirsrv_tmpfs_t:s0      16179200 Jan 13 16:49 __db.002
-rw-------. 1 dirsrv dirsrv system_u:object_r:dirsrv_tmpfs_t:s0     161677312 Jan 13 16:50 __db.003
-rw-------. 1 dirsrv dirsrv unconfined_u:object_r:dirsrv_tmpfs_t:s0        51 Jan 13 16:49 DBVERSION

I also tried re-running `dsctl ldif2db` and the contexts remained the same, reinforcing that it doesn't seem to be purposely changing them.

So my best guess is that either the context relies on the existence of the instance directory or dscreate is fixing the context as part of its configuration.

I also tested the reproducer with a slight twist, replacing an existing IPA installation with one from backup:

1. enable updates-testing, install freeipa-server packages
2. ipa-server-install --domain ipa.test --realm IPA.TEST -a $PASSWORD -p $PASSWORD -U
3. ipa-backup
4. ipa-server-install --uninstall -U
5. ipa-server-install <same options as step 2>
6. ipa-restore /path/to/backup/from/step/3

In this case the contexts on /dev/shm are correct, probably because the /dev/shm instance exists already with the right SELinux context.

dscreate does call restorecon, and we can make dsctl bak2db call it as well.  My issue is that I need to reproduce this outside of IPA so our own QE team can verify it.  Did I miss a step in comment 14 ?

I think the equivalent would be removing /dev/shm/* before calling db2ldif.

(In reply to Rob Crittenden from comment #17)
> I think the equivalent would be removing /dev/shm/* before calling db2ldif.

Ok so "IPA restore" is restoring an entire instance FS layout via a tarball.  Meaning there is no underlying DS instance to begin with.  So dscreate is never used, it just does this untarring and then bak2db and ldif2db are called?

Also why do a bak2db, and then an ldif import?  Seems redundant.

Prior to doing the restore the stored LDIF is re-parsed and any RUV data is stripped out of it. Doing a full restore in IPA requires a database reset on any other replicated servers.

Rob, could you test this DS fedora copr build (f34/35):

https://copr.fedorainfracloud.org/coprs/build/3196603

It has a fix in dsctl that will attempt to relabel the db-home-dir (/dev/shm/slapd-INST) after a successful bak2db.

Looks like I was able to reproduce this by deleting /dev/shm/slapd-INST before doing the bak2db:

[18/Jan/2022:13:25:11.734193584 -0500] - ERR - bdb_version_write - Could not open file "/dev/shm/slapd-localhost/DBVERSION" for writing Netscape Portable Runtime -5966 (Access Denied.)
[18/Jan/2022:13:25:11.739696008 -0500] - ERR - libdb - /dev/shm/slapd-localhost: Permission denied
[18/Jan/2022:13:25:11.742553445 -0500] - CRIT - bdb_start - Opening database environment (/dev/shm/slapd-localhost) failed. err=13: Unexpected dbimpl error code
[18/Jan/2022:13:25:11.748587854 -0500] - ERR - ldbm_back_start - Failed to init database, err=13 Unexpected dbimpl error code


With my test fix (copr build) it works in this scenario.  It would be nice to IAP to verify it before moving forward with the fix.

This COPR build below does a restorecon after a bak2db and ldif2db.  In my testing when I remove /dev/shm this build works and correctly updates /dev/shm/slapd-INSTANCE selinux label.  I was getting an almost identical failure as the IPA logs showed, and I was able to fix it.  However this build (fix in dsctl/python-lib389) in IPA tests shows that restorecon does NOT update the label of /dev/shm/slapd-INSTANCE.  I do not know enough about selinux to explain this behavior.  DS (ns-slapd) can not call restorecon itself because it runs as user "dirsrv' and does not have permission to execute /usr/sbin/restorecon.  

Perhaps IPA's restore script should run restorecon on /dev/shm ?  It would be worth testing at least, but I don't know what else we can do on the DS side.

https://copr.fedorainfracloud.org/coprs/build/3196912

The last thing DS can try is seeing if we can use systemd to call restorecon at server startup.  We will try this next...

you can use tmpfiles.d functionality to force relabeling:

(from tmpfiles.d man page):

       z
           Adjust the access mode, user and group ownership, and restore the SELinux security context of a file or directory, if it exists. Lines of this type accept shell-style globs in place of normal path names. Does not follow symlinks.

       Z
           Recursively set the access mode, user and group ownership, and restore the SELinux security context of a file or directory if it exists, as well as of its subdirectories and the files contained therein (if applicable). Lines of this
           type accept shell-style globs in place of normal path names. Does not follow symlinks.


Define a file and then use systemd-tmpfiles during dirsrv service startup? It might require running systemd-tmpfiles in a separate service file and depend on it.

(In reply to Alexander Bokovoy from comment #26)
> you can use tmpfiles.d functionality to force relabeling:
> 
> (from tmpfiles.d man page):
> 
>        z
>            Adjust the access mode, user and group ownership, and restore the
> SELinux security context of a file or directory, if it exists. Lines of this
> type accept shell-style globs in place of normal path names. Does not follow
> symlinks.
> 
>        Z
>            Recursively set the access mode, user and group ownership, and
> restore the SELinux security context of a file or directory if it exists, as
> well as of its subdirectories and the files contained therein (if
> applicable). Lines of this
>            type accept shell-style globs in place of normal path names. Does
> not follow symlinks.
> 
> 
> Define a file and then use systemd-tmpfiles during dirsrv service startup?
> It might require running systemd-tmpfiles in a separate service file and
> depend on it.

I don't know much about tmpfiles/systemd and trying update /etc/tmpfiles.d/dirsrv/slapd-localhost.conf does not update db_home_dir at server startup:

[root@fedora retrocl]# cat /etc/tmpfiles.d/dirsrv-localhost.conf 
d /run/dirsrv 0770 dirsrv dirsrv
d /run/lock/dirsrv/ 0770 dirsrv dirsrv
d /run/lock/dirsrv/slapd-localhost 0770 dirsrv dirsrv
z /dev/shm/slapd-localhost 0770 dirsrv dirsrv
z /dev/shm/slapd-localhost/* 0600 dirsrv dirsrv


I'll keep looking into this, but I might need help from someone more familiar with this stuff...

Ok, I got this working using an ExecStartPre script in the service file.  I don't know if it will work in the IPA case, but I'll generate a new COPR test build...

Upstream ticket:

https://github.com/389ds/389-ds-base/issues/5127

Fixed upstream:

https://github.com/389ds/389-ds-base/issues/5127

FEDORA-2022-7f62add497 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-7f62add497

FEDORA-2022-7f62add497 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-7f62add497`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-7f62add497

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Tested the bodhi update on fedora 35 (389-ds-base-2.0.13-1.fc35.x86_64), the backup/uninstall/restore scenario works well. I added karma to the bodhi update.

FEDORA-2022-5aee32fbab has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.