Bug 1518599

+++ This bug was initially created as a clone of Bug #1518253 +++

Description of problem:

engine-setup fails with:

[ERROR] Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql'

audit log has:

type=AVC msg=audit(1511854191.913:1057): avc:  denied  { getattr } for  pid=14752 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0" ino=396983 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file

We run this upgrade on CI, and it works there, so there is obviously something specific to the machine this happened on. Not sure what exactly.

Comparing to a machine where it does work, I saw this difference, when running 'semanage fcontext --list': On the good machine, I have:

==========================================
SELinux Local fcontext Equivalence 

/opt/rh/rh-postgresql95/root = /
/var/opt/rh/rh-postgresql95 = /var
/etc/opt/rh/rh-postgresql95 = /etc
==========================================

This is missing on the bad machine.

This seems to be handled by the contents of the file:

selinux/targeted/contexts/files/file_contexts.subs

which is empty on the bad machine, and on the good one has:
==========================================
/opt/rh/rh-postgresql95/root /
/var/opt/rh/rh-postgresql95 /var
/etc/opt/rh/rh-postgresql95 /etc
==========================================

It seems to be created by the postinstall scriptlet of the rpm package rh-postgresql95-runtime. Based on this, I verified the following workaround:

yum reinstall rh-postgresql95-runtime

Based on logs of the bad machine, and the timestamp of the above file, it seems to have been broken by the systemd one-time service selinux-policy-migrate-local-changes. But it did not output anything to the log except for Starting/Started, so can't be sure.

Version-Release number of selected component (if applicable):

Current master

How reproducible:
Not sure

Steps to Reproduce:
1. Have a pre-postgresql9.5 setup. Current was old master (4.2), not sure it affects also 4.1
2. Try to upgrade as usual - yum update setup packages, engine-setup
3.

Actual results:
Fails as above

Expected results:
Succeeds

Additional info:

Workaround:

yum reinstall rh-postgresql95-runtime

and try again.

Opening this bug for reference, for now. Once we get more reports and/or find the root cause, we might close, move to selinux, apply some workaround during engine-setup, or something else.

--- Additional comment from Edward Haas on 2017-11-28 15:11 CET ---

Split by dd into two parts.

--- Additional comment from Edward Haas on 2017-11-28 15:14 CET ---

Split by dd into two parts.

--- Additional comment from Sandro Bonazzola on 2017-11-28 15:16:01 CET ---

Didi please get rh-postgresql95 maintainer involved, maybe some race condition in the order packages are installed due to some missing dep within rh-postgresql95 deptree.

--- Additional comment from Yedidyah Bar David on 2017-11-28 15:27:28 CET ---

Adding selinux-policy and rh-postgresql95 maintainers.

Pavel and Lukas, any of this makes sense?

--- Additional comment from Pavel Raiskup on 2017-11-28 15:39:55 CET ---

(In reply to Yedidyah Bar David from comment #0)
> type=AVC msg=audit(1511854191.913:1057): avc:  denied  { getattr } for 
> pid=14752 comm="postgres"
> path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0"
> ino=396983 scontext=system_u:system_r:postgresql_t:s0
> tcontext=unconfined_u:object_r:var_t:s0 tclass=file

The postgresql.conf file should be of type postgresql_db_t.  If it is not
we need to find the reason why it is not.

> We run this upgrade on CI, and it works there, so there is obviously
> something specific to the machine this happened on. Not sure what exactly.

Do you have SELinux enabled in the CI?

> [...]
> It seems to be created by the postinstall scriptlet of the rpm package
> rh-postgresql95-runtime. Based on this, I verified the following workaround:
> 
> yum reinstall rh-postgresql95-runtime

I would guess the context is OK after postgresql installation,
and db initialization;  and that engine-setup changes the context later.
That script should be analyzed.

--- Additional comment from Yedidyah Bar David on 2017-11-28 16:22:51 CET ---

(In reply to Pavel Raiskup from comment #5)
> (In reply to Yedidyah Bar David from comment #0)
> > type=AVC msg=audit(1511854191.913:1057): avc:  denied  { getattr } for 
> > pid=14752 comm="postgres"
> > path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0"
> > ino=396983 scontext=system_u:system_r:postgresql_t:s0
> > tcontext=unconfined_u:object_r:var_t:s0 tclass=file
> 
> The postgresql.conf file should be of type postgresql_db_t.  If it is not
> we need to find the reason why it is not.
> 
> > We run this upgrade on CI, and it works there, so there is obviously
> > something specific to the machine this happened on. Not sure what exactly.
> 
> Do you have SELinux enabled in the CI?

Yes

> 
> > [...]
> > It seems to be created by the postinstall scriptlet of the rpm package
> > rh-postgresql95-runtime. Based on this, I verified the following workaround:
> > 
> > yum reinstall rh-postgresql95-runtime
> 
> I would guess the context is OK after postgresql installation,
> and db initialization;  and that engine-setup changes the context later.
> That script should be analyzed.

As I wrote above, and discussed in private, engine-setup does not change the label on that file. It copies it from /var/lib/pgsql/data to the /var/opt location using python's shutil.copy2.

Adding also needinfo on Petr, who appears as the author of the script that is ran by selinux-policy-migrate-local-changes, selinux-policy-migrate-local-changes.sh. Petr - any idea?

--- Additional comment from Petr Lautrbach on 2017-11-28 21:27:39 CET ---

selinux-policy-migrate-local-changes.sh script should be run only once after a system is upgraded to rhel-7.3 or from SELinux userspace release < 2.4 to 2.5.

The only place where it touches file_contexts.subs is when it runs:

/usr/sbin/semanage export | /usr/sbin/semanage import

This generally should not fail.

At the same time there is the following line in selinux-policy.spec file which should prevent overwriting this file during selinux-policy update:

%config(noreplace) %{_sysconfdir}/selinux/%1/contexts/files/file_contexts.subs

So it looks like something else removed/erased file_contexts.subs. 

Are you able to reproduce it in a way that you have working setup and do something which results in empty file_contexts.subs ?

--- Additional comment from Yedidyah Bar David on 2017-11-29 07:45:02 CET ---

(In reply to Petr Lautrbach from comment #7)
> selinux-policy-migrate-local-changes.sh script should be run only once after
> a system is upgraded to rhel-7.3 or from SELinux userspace release < 2.4 to
> 2.5.
> 
> The only place where it touches file_contexts.subs is when it runs:
> 
> /usr/sbin/semanage export | /usr/sbin/semanage import
> 
> This generally should not fail.
> 
> At the same time there is the following line in selinux-policy.spec file
> which should prevent overwriting this file during selinux-policy update:
> 
> %config(noreplace)
> %{_sysconfdir}/selinux/%1/contexts/files/file_contexts.subs
> 
> So it looks like something else removed/erased file_contexts.subs. 

Perhaps, but I really have no idea what. Also, the timestamp of that file is ~ 1 second before selinux-policy-migrate-local-changes was "Started" (finished).

> 
> Are you able to reproduce it in a way that you have working setup and do
> something which results in empty file_contexts.subs ?

Didn't try yet.
I have a VM snapshotted and cloned from a machine in a state that has this file zeroed, I can give you access to it for further analysis. The attached sosreport is from the original machine.