RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1581551 - RHEL 7.5: cannot ssh to host after updating selinux-policy and selinux-policy-targeted pkgs to version 3.13.1-193.el7 with kernel version 3.10.0-889.el7
Summary: RHEL 7.5: cannot ssh to host after updating selinux-policy and selinux-policy...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: selinux-policy
Version: 7.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Lukas Vrabec
QA Contact: Milos Malik
URL:
Whiteboard:
: 1581858 1583085 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-23 04:53 UTC by Walid A.
Modified: 2018-10-30 10:04 UTC (History)
11 users (show)

Fixed In Version: selinux-policy-3.13.1-199.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-30 10:04:11 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3111 0 None None None 2018-10-30 10:04:48 UTC

Description Walid A. 2018-05-23 04:53:42 UTC 
Description of problem:
When we run yum update on a RHEL 7.5 host in AWS EC2, starting with kernel kernel.x86_64 version 3.10.0-862.el7 and selinux-policy.noarch and selinux-policy-targeted.noarch version 3.13.1-192.el7, yum update is successful. The kernel is updated to version 3.10.0-889.el7, and selinux-policy and selinux-policy-targeted packages both get updated to version 3.13.1-193.el7.  But after reboot, I cannot ssh to the instance and it will fail the EC2 Status Check.  I don't have console access to these instances but I have collected the terminal session logs and from journal and audit.log files..

When I exclude both packages selinux-policy and selinux-policy-targeted from yum update, the kernel gets updated successfully to version 3.10.0-889.el7 and along with other packages, and the host can be rebooted successfully and we can ssh to it.  

As soon as I yum update selinux-policy and selinux-policy-targeted to version 3.13.1-193.el7 (only one available from rhel-7-next repos), if I exist of my current ssh session, I cannot ssh back.  I get Authetnicaltion failed.  Also if I reboot after the yum update, I am not able to ssh to the host after reboot.



Version-Release number of selected component (if applicable):
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
# Before yum update:
- selinux-policy and selinux-policy-targeted pkgs on version 3.13.1-192.el7 
- kernel on version 3.10.0-862.el7
# After yum update:
- selinux-policy and selinux-policy-targeted pkgs on version 3.13.1-193.el7 
- kernel on version 3.10.0-889.el7

How reproducible:
Always

Steps to Reproduce:
1. Start with a RHEL 7.5 instance on AWS on kernel version 3.10.0-862.el7, and 3.13.1-192.el7
2. yum update kernel
3. kernel will update to version 3.10.0-889.el7
4.  yum update selinux-policy selinux-policy-targeted 
  selinux-policy selinux-policy-targeted get updated to version 3.13.1-193.el7
5. Exist ssh session,  You can no longer ssh back.  Or if you reboot the updated host, with "shutdown -r now" or "reboot", you can no longer ssh back to the host anymore after reboot.

Actual results:
Cannot ssh to the host anymore after reboot

Expected results:
Should be able to ssh to host after yum update followed by reboot

Additional info:
Logs from yum update, journal, syslog, and audit.log were captured from the terminal session before we lost ssh access.  Please see next comment for link to logs.
Comment 3 Milos Malik 2018-05-23 06:45:11 UTC 
My guess is that { map } AVCs appear on the machine where the selinux-policy upgrade happened.

Could you login to the machine before selinux-policy upgrade, stay logged in and collect SELinux denials that appear after the upgrade?

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today
Comment 4 Milos Malik 2018-05-23 07:13:27 UTC 
Following SELinux denials appeared on my machine after kernel + selinux-policy update and reboot:
----
type=PROCTITLE msg=audit(05/23/2018 03:09:29.187:23) : proctitle=/sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5fb06bd 
type=SYSCALL msg=audit(05/23/2018 03:09:29.187:23) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x55f65dcb34ce a1=0x55f65dd6a920 a2=0x55f65dcad710 a3=0x7fff2cdd9760 items=0 ppid=941 pid=959 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=dhclient exe=/usr/sbin/dhclient subj=system_u:system_r:dhcpc_t:s0 key=(null) 
type=AVC msg=audit(05/23/2018 03:09:29.187:23) : avc:  denied  { map } for  pid=959 comm=dhclient path=/usr/sbin/dhclient dev="vda1" ino=6776920 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:dhcpc_exec_t:s0 tclass=file permissive=1 
----
type=PROCTITLE msg=audit(05/23/2018 03:09:34.563:49) : proctitle=pickup -l -t unix -u 
type=SYSCALL msg=audit(05/23/2018 03:09:34.563:49) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x56142a90e480 a1=0x56142a9172c0 a2=0x56142a913680 a3=0xffffffff items=0 ppid=1655 pid=1659 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=pickup exe=/usr/libexec/postfix/pickup subj=system_u:system_r:postfix_pickup_t:s0 key=(null) 
type=AVC msg=audit(05/23/2018 03:09:34.563:49) : avc:  denied  { map } for  pid=1659 comm=pickup path=/usr/libexec/postfix/pickup dev="vda1" ino=256669 scontext=system_u:system_r:postfix_master_t:s0 tcontext=system_u:object_r:postfix_pickup_exec_t:s0 tclass=file permissive=1 
----
type=PROCTITLE msg=audit(05/23/2018 03:09:34.560:53) : proctitle=qmgr -l -t unix -u 
type=SYSCALL msg=audit(05/23/2018 03:09:34.560:53) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x56142a917720 a1=0x56142a917b20 a2=0x56142a913680 a3=0xffffffff items=0 ppid=1655 pid=1660 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=qmgr exe=/usr/libexec/postfix/qmgr subj=system_u:system_r:postfix_qmgr_t:s0 key=(null) 
type=AVC msg=audit(05/23/2018 03:09:34.560:53) : avc:  denied  { map } for  pid=1660 comm=qmgr path=/usr/libexec/postfix/qmgr dev="vda1" ino=258785 scontext=system_u:system_r:postfix_master_t:s0 tcontext=system_u:object_r:postfix_qmgr_exec_t:s0 tclass=file permissive=1 
----
type=PROCTITLE msg=audit(05/23/2018 03:10:03.239:73) : proctitle=/usr/sbin/unix_chkpwd root chkexpiry 
type=SYSCALL msg=audit(05/23/2018 03:10:03.239:73) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x7f513b2f634d a1=0x7ffe62c16ab0 a2=0x7f513b4f9368 a3=0x2 items=0 ppid=1700 pid=1703 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=unix_chkpwd exe=/usr/sbin/unix_chkpwd subj=system_u:system_r:chkpwd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(05/23/2018 03:10:03.239:73) : avc:  denied  { map } for  pid=1703 comm=unix_chkpwd path=/usr/sbin/unix_chkpwd dev="vda1" ino=6467114 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:chkpwd_exec_t:s0 tclass=file permissive=1 
----

The machine was running in permissive mode, because /boot/grub2/grub.cfg file contains "enforcing=0" on the kernel command line.
Comment 5 Milos Malik 2018-05-23 07:33:01 UTC 
The first SELinux denial in comment#4 is already addressed in BZ#1574383.
I believe the last SELinux denial in comment#4 is the cause why you cannot log in back via ssh.
Comment 6 Milos Malik 2018-05-23 14:43:25 UTC 
Following SELinux denial did not make it to audit.log:

# dmesg | grep type=1400
[    3.227820] type=1400 audit(1527086219.186:3): avc:  denied  { map } for  pid=835 comm="audispd" path="/usr/sbin/audispd" dev="vda1" ino=6842012 scontext=system_u:system_r:auditd_t:s0 tcontext=system_u:object_r:audisp_exec_t:s0 tclass=file permissive=1
#

but it appeared when the machine booted in permissive mode:

# rpm -qa kernel\* selinux-policy\* | sort
kernel-3.10.0-890.el7.x86_64
kernel-debug-devel-3.10.0-890.el7.x86_64
kernel-headers-3.10.0-890.el7.x86_64
kernel-tools-3.10.0-890.el7.x86_64
kernel-tools-libs-3.10.0-890.el7.x86_64
selinux-policy-3.13.1-193.el7.noarch
selinux-policy-devel-3.13.1-193.el7.noarch
selinux-policy-targeted-3.13.1-193.el7.noarch
#
Comment 7 Walid A. 2018-05-23 16:38:03 UTC 
Info for comment 3:

This after upgrading the kernel to 3.10.0-889.el7 and rebooted,
and BEFORE upgrading selinux-policy, selinux-policy-targeted:
-------------------------------------------------------------

# yum list selinux*
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
selinux-policy.noarch                                     3.13.1-192.el7_5.3                             @oso-rhui-rhel-server-releases
selinux-policy-targeted.noarch                            3.13.1-192.el7_5.3                             @oso-rhui-rhel-server-releases
Available Packages
selinux-policy.noarch                                     3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-devel.noarch                               3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-doc.noarch                                 3.13.1-193.el7                                 rhel-7-optional-next          
selinux-policy-minimum.noarch                             3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-mls.noarch                                 3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-sandbox.noarch                             3.13.1-193.el7                                 rhel-7-optional-next          
selinux-policy-targeted.noarch                            3.13.1-193.el7                                 rhel-7-next                   
root@ip-172-31-16-185: ~ # 

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today
<no matches>

# yum list kernel* 
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
kernel.x86_64                                                       3.10.0-862.el7                                      @anaconda/7.5       
kernel.x86_64                                                       3.10.0-888.el7                                      @rhel-7-next        
kernel.x86_64                                                       3.10.0-889.el7                                      @rhel-7-next        
kernel-devel.x86_64                                                 3.10.0-888.el7                                      @rhel-7-next        
kernel-devel.x86_64                                                 3.10.0-889.el7                                      @rhel-7-next        
kernel-headers.x86_64                                               3.10.0-889.el7                                      @rhel-7-next        
kernel-tools.x86_64                                                 3.10.0-889.el7                                      @rhel-7-next        
kernel-tools-libs.x86_64                                            3.10.0-889.el7                                      @rhel-7-next        
Available Packages
kernel-abi-whitelists.noarch                                        3.10.0-889.el7                                      rhel-7-next         
kernel-debug.x86_64                                                 3.10.0-889.el7                                      rhel-7-next         
kernel-debug-devel.x86_64                                           3.10.0-889.el7                                      rhel-7-next         
kernel-doc.noarch                                                   3.10.0-889.el7                                      rhel-7-next         
kernel-tools-libs-devel.x86_64                                      3.10.0-889.el7                                      rhel-7-optional-next

# uname -r
3.10.0-889.el7.x86_64


############ After upgrading selinux-policy and selinux-policy-targeted:

# yum list selinux-policy*
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
selinux-policy.noarch                                               3.13.1-193.el7                                      @rhel-7-next        
selinux-policy-targeted.noarch                                      3.13.1-193.el7                                      @rhel-7-next        
Available Packages
selinux-policy-devel.noarch                                         3.13.1-193.el7                                      rhel-7-next         
selinux-policy-doc.noarch                                           3.13.1-193.el7                                      rhel-7-optional-next
selinux-policy-minimum.noarch                                       3.13.1-193.el7                                      rhel-7-next         
selinux-policy-mls.noarch                                           3.13.1-193.el7                                      rhel-7-next         
selinux-policy-sandbox.noarch                                       3.13.1-193.el7                                      


#### selinux denials AFTER upgrade selinux-policy*

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today
----
type=USER_AVC msg=audit(05/23/2018 16:19:38.829:105) : pid=595 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  received policyload notice (seqno=2)  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?'
Comment 8 Lukas Vrabec 2018-05-24 12:49:45 UTC 
*** Bug 1581858 has been marked as a duplicate of this bug. ***
Comment 9 Jarod Wilson 2018-05-24 17:17:41 UTC 
Was bisecting odd network failures last night myself, and found that it was an selinux kernel change in 3.10.0-375.el7 that started triggering problems for me, got pointed this direction.
Comment 12 Lukas Vrabec 2018-06-03 15:48:01 UTC 
*** Bug 1583085 has been marked as a duplicate of this bug. ***
Comment 17 errata-xmlrpc 2018-10-30 10:04:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3111
Note You need to log in before you can comment on or make changes to this bug.