Description of problem: engine-setup fails with: [ERROR] Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' audit log has: type=AVC msg=audit(1511854191.913:1057): avc: denied { getattr } for pid=14752 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0" ino=396983 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file We run this upgrade on CI, and it works there, so there is obviously something specific to the machine this happened on. Not sure what exactly. Comparing to a machine where it does work, I saw this difference, when running 'semanage fcontext --list': On the good machine, I have: ========================================== SELinux Local fcontext Equivalence /opt/rh/rh-postgresql95/root = / /var/opt/rh/rh-postgresql95 = /var /etc/opt/rh/rh-postgresql95 = /etc ========================================== This is missing on the bad machine. This seems to be handled by the contents of the file: selinux/targeted/contexts/files/file_contexts.subs which is empty on the bad machine, and on the good one has: ========================================== /opt/rh/rh-postgresql95/root / /var/opt/rh/rh-postgresql95 /var /etc/opt/rh/rh-postgresql95 /etc ========================================== It seems to be created by the postinstall scriptlet of the rpm package rh-postgresql95-runtime. Based on this, I verified the following workaround: yum reinstall rh-postgresql95-runtime Based on logs of the bad machine, and the timestamp of the above file, it seems to have been broken by the systemd one-time service selinux-policy-migrate-local-changes. But it did not output anything to the log except for Starting/Started, so can't be sure. Version-Release number of selected component (if applicable): Current master How reproducible: Not sure Steps to Reproduce: 1. Have a pre-postgresql9.5 setup. Current was old master (4.2), not sure it affects also 4.1 2. Try to upgrade as usual - yum update setup packages, engine-setup 3. Actual results: Fails as above Expected results: Succeeds Additional info: Workaround: yum reinstall rh-postgresql95-runtime and try again. Opening this bug for reference, for now. Once we get more reports and/or find the root cause, we might close, move to selinux, apply some workaround during engine-setup, or something else.
Created attachment 1359890 [details] sosreport-edwardh-host-1-part1 Split by dd into two parts.
Created attachment 1359891 [details] sosreport-edwardh-host-1-part2 Split by dd into two parts.
Didi please get rh-postgresql95 maintainer involved, maybe some race condition in the order packages are installed due to some missing dep within rh-postgresql95 deptree.
Adding selinux-policy and rh-postgresql95 maintainers. Pavel and Lukas, any of this makes sense?
(In reply to Yedidyah Bar David from comment #0) > type=AVC msg=audit(1511854191.913:1057): avc: denied { getattr } for > pid=14752 comm="postgres" > path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0" > ino=396983 scontext=system_u:system_r:postgresql_t:s0 > tcontext=unconfined_u:object_r:var_t:s0 tclass=file The postgresql.conf file should be of type postgresql_db_t. If it is not we need to find the reason why it is not. > We run this upgrade on CI, and it works there, so there is obviously > something specific to the machine this happened on. Not sure what exactly. Do you have SELinux enabled in the CI? > [...] > It seems to be created by the postinstall scriptlet of the rpm package > rh-postgresql95-runtime. Based on this, I verified the following workaround: > > yum reinstall rh-postgresql95-runtime I would guess the context is OK after postgresql installation, and db initialization; and that engine-setup changes the context later. That script should be analyzed.
(In reply to Pavel Raiskup from comment #5) > (In reply to Yedidyah Bar David from comment #0) > > type=AVC msg=audit(1511854191.913:1057): avc: denied { getattr } for > > pid=14752 comm="postgres" > > path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0" > > ino=396983 scontext=system_u:system_r:postgresql_t:s0 > > tcontext=unconfined_u:object_r:var_t:s0 tclass=file > > The postgresql.conf file should be of type postgresql_db_t. If it is not > we need to find the reason why it is not. > > > We run this upgrade on CI, and it works there, so there is obviously > > something specific to the machine this happened on. Not sure what exactly. > > Do you have SELinux enabled in the CI? Yes > > > [...] > > It seems to be created by the postinstall scriptlet of the rpm package > > rh-postgresql95-runtime. Based on this, I verified the following workaround: > > > > yum reinstall rh-postgresql95-runtime > > I would guess the context is OK after postgresql installation, > and db initialization; and that engine-setup changes the context later. > That script should be analyzed. As I wrote above, and discussed in private, engine-setup does not change the label on that file. It copies it from /var/lib/pgsql/data to the /var/opt location using python's shutil.copy2. Adding also needinfo on Petr, who appears as the author of the script that is ran by selinux-policy-migrate-local-changes, selinux-policy-migrate-local-changes.sh. Petr - any idea?
selinux-policy-migrate-local-changes.sh script should be run only once after a system is upgraded to rhel-7.3 or from SELinux userspace release < 2.4 to 2.5. The only place where it touches file_contexts.subs is when it runs: /usr/sbin/semanage export | /usr/sbin/semanage import This generally should not fail. At the same time there is the following line in selinux-policy.spec file which should prevent overwriting this file during selinux-policy update: %config(noreplace) %{_sysconfdir}/selinux/%1/contexts/files/file_contexts.subs So it looks like something else removed/erased file_contexts.subs. Are you able to reproduce it in a way that you have working setup and do something which results in empty file_contexts.subs ?
(In reply to Petr Lautrbach from comment #7) > selinux-policy-migrate-local-changes.sh script should be run only once after > a system is upgraded to rhel-7.3 or from SELinux userspace release < 2.4 to > 2.5. > > The only place where it touches file_contexts.subs is when it runs: > > /usr/sbin/semanage export | /usr/sbin/semanage import > > This generally should not fail. > > At the same time there is the following line in selinux-policy.spec file > which should prevent overwriting this file during selinux-policy update: > > %config(noreplace) > %{_sysconfdir}/selinux/%1/contexts/files/file_contexts.subs > > So it looks like something else removed/erased file_contexts.subs. Perhaps, but I really have no idea what. Also, the timestamp of that file is ~ 1 second before selinux-policy-migrate-local-changes was "Started" (finished). > > Are you able to reproduce it in a way that you have working setup and do > something which results in empty file_contexts.subs ? Didn't try yet. I have a VM snapshotted and cloned from a machine in a state that has this file zeroed, I can give you access to it for further analysis. The attached sosreport is from the original machine.
I cloned this bug to selinux-policy. It seems to be some corner case when selinux-policy-3.13.1-102.el7.noarch is updated to 3.13.1-166.el7_4.7.noarch in a particular condition Until it's fixed please use the workaround from
Changing to a Known Issue for now. We might clone and implement a workaround in the code if we get more reports.
Edited doc text a bit. Also perhaps consider adding somewhere 'Under certain conditions', because we got very few reports like this, and do not fully know the details. Thanks.
Thanks for the review Didi. I've added under certain conditions and verified the changes.
INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [No external trackers attached] For more info please contact: infra
Upgrade 4.1 -> 4.2 was sucessfull. I tried it more times on different environments (with selinux-policy-targeted version 3.13.1-102.el7, 3.13.1-166.el7_4.7 and 3.13.1-183.el7). Feel free to reopen the bug if it happens again. verified in ovirt-engine-4.2.1.2-0.1.el7.noarch
This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.
Another case where it happen: https://bugzilla.redhat.com/show_bug.cgi?id=1594615