Bug 1518253
Summary: | engine-setup upgrade to postgresql 9.5 sometimes fails due to missing selinux policy | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [oVirt] ovirt-engine | Reporter: | Yedidyah Bar David <didi> | ||||||
Component: | Setup.Engine | Assignee: | Yedidyah Bar David <didi> | ||||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Lucie Leistnerova <lleistne> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 4.2.0 | CC: | bugs, edwardh, lsvaty, lvrabec, melewis, omachace, plautrba | ||||||
Target Milestone: | ovirt-4.2.1 | Flags: | rule-engine:
ovirt-4.2+
|
||||||
Target Release: | 4.2.1 | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Known Issue | |||||||
Doc Text: |
Under certain conditions an issue with a change in selinux policy, and the script that converts a selinux policy in the old format to the new format of the selinux policy, causes the engine-setup upgrade to postgresql to fail with the error '[ERROR] Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql'. The log in /var/log/messages shows 'postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied'. To prevent this, reinstall the rh-postgresql95-runtime package by running 'yum reinstall rh-postgresql95-runtime', then run engine-setup again.
|
Story Points: | --- | ||||||
Clone Of: | |||||||||
: | 1518599 (view as bug list) | Environment: | |||||||
Last Closed: | 2018-02-12 11:48:32 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | Integration | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Yedidyah Bar David
2017-11-28 13:51:13 UTC
Created attachment 1359890 [details]
sosreport-edwardh-host-1-part1
Split by dd into two parts.
Created attachment 1359891 [details]
sosreport-edwardh-host-1-part2
Split by dd into two parts.
Didi please get rh-postgresql95 maintainer involved, maybe some race condition in the order packages are installed due to some missing dep within rh-postgresql95 deptree. Adding selinux-policy and rh-postgresql95 maintainers. Pavel and Lukas, any of this makes sense? (In reply to Yedidyah Bar David from comment #0) > type=AVC msg=audit(1511854191.913:1057): avc: denied { getattr } for > pid=14752 comm="postgres" > path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0" > ino=396983 scontext=system_u:system_r:postgresql_t:s0 > tcontext=unconfined_u:object_r:var_t:s0 tclass=file The postgresql.conf file should be of type postgresql_db_t. If it is not we need to find the reason why it is not. > We run this upgrade on CI, and it works there, so there is obviously > something specific to the machine this happened on. Not sure what exactly. Do you have SELinux enabled in the CI? > [...] > It seems to be created by the postinstall scriptlet of the rpm package > rh-postgresql95-runtime. Based on this, I verified the following workaround: > > yum reinstall rh-postgresql95-runtime I would guess the context is OK after postgresql installation, and db initialization; and that engine-setup changes the context later. That script should be analyzed. (In reply to Pavel Raiskup from comment #5) > (In reply to Yedidyah Bar David from comment #0) > > type=AVC msg=audit(1511854191.913:1057): avc: denied { getattr } for > > pid=14752 comm="postgres" > > path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="dm-0" > > ino=396983 scontext=system_u:system_r:postgresql_t:s0 > > tcontext=unconfined_u:object_r:var_t:s0 tclass=file > > The postgresql.conf file should be of type postgresql_db_t. If it is not > we need to find the reason why it is not. > > > We run this upgrade on CI, and it works there, so there is obviously > > something specific to the machine this happened on. Not sure what exactly. > > Do you have SELinux enabled in the CI? Yes > > > [...] > > It seems to be created by the postinstall scriptlet of the rpm package > > rh-postgresql95-runtime. Based on this, I verified the following workaround: > > > > yum reinstall rh-postgresql95-runtime > > I would guess the context is OK after postgresql installation, > and db initialization; and that engine-setup changes the context later. > That script should be analyzed. As I wrote above, and discussed in private, engine-setup does not change the label on that file. It copies it from /var/lib/pgsql/data to the /var/opt location using python's shutil.copy2. Adding also needinfo on Petr, who appears as the author of the script that is ran by selinux-policy-migrate-local-changes, selinux-policy-migrate-local-changes.sh. Petr - any idea? selinux-policy-migrate-local-changes.sh script should be run only once after a system is upgraded to rhel-7.3 or from SELinux userspace release < 2.4 to 2.5. The only place where it touches file_contexts.subs is when it runs: /usr/sbin/semanage export | /usr/sbin/semanage import This generally should not fail. At the same time there is the following line in selinux-policy.spec file which should prevent overwriting this file during selinux-policy update: %config(noreplace) %{_sysconfdir}/selinux/%1/contexts/files/file_contexts.subs So it looks like something else removed/erased file_contexts.subs. Are you able to reproduce it in a way that you have working setup and do something which results in empty file_contexts.subs ? (In reply to Petr Lautrbach from comment #7) > selinux-policy-migrate-local-changes.sh script should be run only once after > a system is upgraded to rhel-7.3 or from SELinux userspace release < 2.4 to > 2.5. > > The only place where it touches file_contexts.subs is when it runs: > > /usr/sbin/semanage export | /usr/sbin/semanage import > > This generally should not fail. > > At the same time there is the following line in selinux-policy.spec file > which should prevent overwriting this file during selinux-policy update: > > %config(noreplace) > %{_sysconfdir}/selinux/%1/contexts/files/file_contexts.subs > > So it looks like something else removed/erased file_contexts.subs. Perhaps, but I really have no idea what. Also, the timestamp of that file is ~ 1 second before selinux-policy-migrate-local-changes was "Started" (finished). > > Are you able to reproduce it in a way that you have working setup and do > something which results in empty file_contexts.subs ? Didn't try yet. I have a VM snapshotted and cloned from a machine in a state that has this file zeroed, I can give you access to it for further analysis. The attached sosreport is from the original machine. I cloned this bug to selinux-policy. It seems to be some corner case when selinux-policy-3.13.1-102.el7.noarch is updated to 3.13.1-166.el7_4.7.noarch in a particular condition Until it's fixed please use the workaround from Changing to a Known Issue for now. We might clone and implement a workaround in the code if we get more reports. Edited doc text a bit. Also perhaps consider adding somewhere 'Under certain conditions', because we got very few reports like this, and do not fully know the details. Thanks. Thanks for the review Didi. I've added under certain conditions and verified the changes. INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [No external trackers attached] For more info please contact: infra INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason: [No external trackers attached] For more info please contact: infra Upgrade 4.1 -> 4.2 was sucessfull. I tried it more times on different environments (with selinux-policy-targeted version 3.13.1-102.el7, 3.13.1-166.el7_4.7 and 3.13.1-183.el7). Feel free to reopen the bug if it happens again. verified in ovirt-engine-4.2.1.2-0.1.el7.noarch This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. Another case where it happen: https://bugzilla.redhat.com/show_bug.cgi?id=1594615 |