Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Docker can't be started because rhel-push-plugin.sock can't be started after upgraded. If setenfore 0, the docker can be started.
At least the issue exsit in branch '7.2.internal.0.209, 7.2.5(5b82b4035f)'.
No such issue if there isn't openshift installed on atomic host.
Version-Release number of selected component (if applicable):
atomic-host (docker 1.8) upgrade to atomic-host (docker 1.10)
How reproducible:
always on my Env
Steps to Reproduce:
1) install openshift 3.1 on atomic-openshift
2) upgrade atomic host to 7.2.internal.0.209 or 7.2.5
3) check the docker/rhel-push-plugin.sock status
4) systemctl start rhel-push-plugin.sock and check logs
Actual results:
4) -bash-4.2# journalctl -xe
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-1.scope has finished starting up.
--
-- The start-up result is done.
Jun 15 07:14:02 host4master.example.com systemd-logind[717]: New session 1 of user root.
-- Subject: A new session 1 has been created for user root
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A new session with the ID 1 has been created for the user root.
--
-- The leading process of the session is 11210.
Jun 15 07:14:02 host4master.example.com systemd[1]: Starting Session 1 of user root.
-- Subject: Unit session-1.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/li
stinfo/systemd-devel
--
-- Unit session-1.scope has begun starting up.
Jun 15 07:14:02 host4master.example.com sshd[11210]: pam_unix(sshd:session): session opened for user root by (uid=0)
Jun 15 07:14:02 host4master.example.com sshd[11210]: pam_lastlog(sshd:session): unable to open /var/log/lastlog: No such file or directory
Jun 15 07:14:02 host4master.example.com sshd[11213]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
Jun 15 07:14:02 host4master.example.com sshd[11213]: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
Jun 15 07:15:19 host4master.example.com polkitd[918]: Registered Authentication Agent for unix-process:11242:16929 (system bus name :1.9 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedeskt
Jun 15 07:15:19 host4master.example.com kernel: type=1400 audit(1465974919.614:9): avc: denied { mac_admin } for pid=1 comm="systemd" capability=33 scontext=system_u:system_r:init_t:s0 tcontext=system_u:syst
Jun 15 07:15:19 host4master.example.com kernel: type=1401 audit(1465974919.617:10): op=fscreate invalid_context="system_u:object_r:docker_plugin_var_run_t:s0"
Jun 15 07:15:19 host4master.example.com systemd[1]: Failed to set SELinux security context system_u:object_r:docker_plugin_var_run_t:s0 for /run/docker/plugins: Invalid argument
Jun 15 07:15:19 host4master.example.com kernel: type=1400 audit(1465974919.618:11): avc: denied { mac_admin } for pid=1 comm="systemd" capability=33 scontext=system_u:system_r:init_t:s0 tcontext=system_u:sys
Jun 15 07:15:19 host4master.example.com kernel: type=1401 audit(1465974919.621:12): op=fscreate invalid_context="system_u:object_r:docker_plugin_var_run_t:s0"
Jun 15 07:15:19 host4master.example.com systemd[1]: Failed to set SELinux security context system_u:object_r:docker_plugin_var_run_t:s0 for /run/docker/plugins/rhel-push-plugin.sock: Invalid argument
Jun 15 07:15:19 host4master.example.com systemd[1]: rhel-push-plugin.socket failed to listen on sockets: Invalid argument
Jun 15 07:15:19 host4master.example.com systemd[1]: Failed to listen on Docker Block RHEL push plugin Socket for the API.
-- Subject: Unit rhel-push-plugin.socket has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rhel-push-plugin.socket has failed.
--
-- The result is failed.
Expected Result:
Atomic host can be upgrade to docker 1.10
Additional info:
Host status after upgrade
-bash-4.2# atomic host status
TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC
* 2016-06-13 17:33:11 7.2.5 5b82b4035f rhel-atomic-host rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
2016-05-06 05:57:30 7.2.4 b060975ce3 rhel-atomic-host rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
-bash-4.2# rpm-ostree status
TIMESTAMP (UTC) VERSION ID OSNAME REFSPEC
* 2016-06-09 21:40:48 7.2.internal.0.209 f5749e8add rhel-atomic-host rhel-atomic-host-ostree:rhel-atomic-host/7.2/x86_64/autobrew/buildmaster
2016-03-01 01:35:21 7.2.2-2 8b2cf24b42 rhel-atomic-host rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
We have been hitting this bug when upgrading RHEL Atomic to versions 7.2.5 and newer on a few of our hosts (not all of them fortunately).
After the upgrade docker.service wouldn't start due to failed dependencies, while rhel-push-plugin.socket was unable to create the socket because of SELinux failing at 'Invalid argument'.
Example for trying to use the 'docker_plugin_var_run_t':
# mkdir /run/docker/plugins
# restorecon -vvFR /run/docker/plugins
restorecon reset /run/docker/plugins context unconfined_u:object_r:docker_var_run_t:s0->system_u:object_r:docker_plugin_var_run_t:s0
restorecon set context /run/docker/plugins->system_u:object_r:docker_plugin_var_run_t:s0 failed:'Invalid argument'
This has been solved by issuing the following:
# semodule -e docker
The module was enabled before, but for some reason using 'docker_plugin_var_run_t' was not allowed.
The mentioned hosts had SELinux in enforcing mode, but might've been switched to permissive at some point in the past.
Comment 11Jakub Rączkowski
2017-01-28 13:09:46 UTC
I have hit the same bug with 7.3.1 (upgraded from previous releases), I've fixed the installation by reenabling the newly introduced module:
# semodule -e container
Don't ever run `restorecon /` on Atomic Host - you will break it (since SELinux doesn't know about /sysroot where the snapshots are stored).
I think the issue here is there's an interaction bug between SELinux and ostree - if you ever change the policy (e.g. via seboolan), after that you will need to manually rerun `semodule -B` after rebooting on upgrades. Yes, this is a nasty bug and we should prioritize fixing it.
Can anyone who is hitting this issue try this:
sha256sum /{usr/,}etc/selinux/targeted/policy/policy.30
If the policies are different, you can reset back via:
cp /{usr/,}etc/selinux/targeted/policy/policy.30
This will undo any local policy changes, and should ensure you get updated policy versions.
We should label /sysroot <<none>>
Which would prevent restorecon from doing anything there.
The reason for the restorecon /run is that docker running as unconfined_service_t would have created the socket as var_run_t, when you fixed the container-selinux install it now runs as container_runtime_t which would create the socket as container_runtime_var_run_t.
Comment 19Jakub Rączkowski
2017-02-25 20:24:26 UTC
Colin - here are the checksums you've requested:
# sha256sum /{usr/,}etc/selinux/targeted/policy/policy.30
90d85b78150ad6e60a5be4789b8165f2b6d6199d7ae7fbe1755ad19cbe2959be /usr/etc/selinux/targeted/policy/policy.30
b24a554085291ff2e238b7a8e619e57524f3b694dc45bc2fb4f85836dd24c759 /etc/selinux/targeted/policy/policy.30
This is for version 7.3.2-1, though we've added a custom SELinux module due to a bug registered as 1358702 in Red Hat (Selinux prevents access to /etc/hosts in Atomic Host 7.2.5) with the following policy:
module docker-var-ro 1.0;
require {
type svirt_lxc_net_t;
type docker_var_lib_t;
class file { ioctl read getattr open };
}
#============= svirt_lxc_net_t ==============
allow svirt_lxc_net_t docker_var_lib_t:file { ioctl read getattr open };
The above is probably inaccurate as of the introduction of the 'container' module.