Bug 1581551

Summary: RHEL 7.5: cannot ssh to host after updating selinux-policy and selinux-policy-targeted pkgs to version 3.13.1-193.el7 with kernel version 3.10.0-889.el7
Product: Red Hat Enterprise Linux 7 Reporter: Walid A. <wabouham>
Component: selinux-policyAssignee: Lukas Vrabec <lvrabec>
Status: CLOSED ERRATA QA Contact: Milos Malik <mmalik>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.5CC: ekuric, jarod, lslebodn, lvrabec, mgrepl, mifiedle, mmalik, plautrba, prarit, ssekidde, wabouham
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: selinux-policy-3.13.1-199.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-30 10:04:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Walid A. 2018-05-23 04:53:42 UTC
Description of problem:
When we run yum update on a RHEL 7.5 host in AWS EC2, starting with kernel kernel.x86_64 version 3.10.0-862.el7 and selinux-policy.noarch and selinux-policy-targeted.noarch version 3.13.1-192.el7, yum update is successful. The kernel is updated to version 3.10.0-889.el7, and selinux-policy and selinux-policy-targeted packages both get updated to version 3.13.1-193.el7.  But after reboot, I cannot ssh to the instance and it will fail the EC2 Status Check.  I don't have console access to these instances but I have collected the terminal session logs and from journal and audit.log files..

When I exclude both packages selinux-policy and selinux-policy-targeted from yum update, the kernel gets updated successfully to version 3.10.0-889.el7 and along with other packages, and the host can be rebooted successfully and we can ssh to it.  

As soon as I yum update selinux-policy and selinux-policy-targeted to version 3.13.1-193.el7 (only one available from rhel-7-next repos), if I exist of my current ssh session, I cannot ssh back.  I get Authetnicaltion failed.  Also if I reboot after the yum update, I am not able to ssh to the host after reboot.



Version-Release number of selected component (if applicable):
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
# Before yum update:
- selinux-policy and selinux-policy-targeted pkgs on version 3.13.1-192.el7 
- kernel on version 3.10.0-862.el7
# After yum update:
- selinux-policy and selinux-policy-targeted pkgs on version 3.13.1-193.el7 
- kernel on version 3.10.0-889.el7

How reproducible:
Always

Steps to Reproduce:
1. Start with a RHEL 7.5 instance on AWS on kernel version 3.10.0-862.el7, and 3.13.1-192.el7
2. yum update kernel
3. kernel will update to version 3.10.0-889.el7
4.  yum update selinux-policy selinux-policy-targeted 
  selinux-policy selinux-policy-targeted get updated to version 3.13.1-193.el7
5. Exist ssh session,  You can no longer ssh back.  Or if you reboot the updated host, with "shutdown -r now" or "reboot", you can no longer ssh back to the host anymore after reboot.

Actual results:
Cannot ssh to the host anymore after reboot

Expected results:
Should be able to ssh to host after yum update followed by reboot

Additional info:
Logs from yum update, journal, syslog, and audit.log were captured from the terminal session before we lost ssh access.  Please see next comment for link to logs.

Comment 3 Milos Malik 2018-05-23 06:45:11 UTC
My guess is that { map } AVCs appear on the machine where the selinux-policy upgrade happened.

Could you login to the machine before selinux-policy upgrade, stay logged in and collect SELinux denials that appear after the upgrade?

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today

Comment 4 Milos Malik 2018-05-23 07:13:27 UTC
Following SELinux denials appeared on my machine after kernel + selinux-policy update and reboot:
----
type=PROCTITLE msg=audit(05/23/2018 03:09:29.187:23) : proctitle=/sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5fb06bd 
type=SYSCALL msg=audit(05/23/2018 03:09:29.187:23) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x55f65dcb34ce a1=0x55f65dd6a920 a2=0x55f65dcad710 a3=0x7fff2cdd9760 items=0 ppid=941 pid=959 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=dhclient exe=/usr/sbin/dhclient subj=system_u:system_r:dhcpc_t:s0 key=(null) 
type=AVC msg=audit(05/23/2018 03:09:29.187:23) : avc:  denied  { map } for  pid=959 comm=dhclient path=/usr/sbin/dhclient dev="vda1" ino=6776920 scontext=system_u:system_r:NetworkManager_t:s0 tcontext=system_u:object_r:dhcpc_exec_t:s0 tclass=file permissive=1 
----
type=PROCTITLE msg=audit(05/23/2018 03:09:34.563:49) : proctitle=pickup -l -t unix -u 
type=SYSCALL msg=audit(05/23/2018 03:09:34.563:49) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x56142a90e480 a1=0x56142a9172c0 a2=0x56142a913680 a3=0xffffffff items=0 ppid=1655 pid=1659 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=pickup exe=/usr/libexec/postfix/pickup subj=system_u:system_r:postfix_pickup_t:s0 key=(null) 
type=AVC msg=audit(05/23/2018 03:09:34.563:49) : avc:  denied  { map } for  pid=1659 comm=pickup path=/usr/libexec/postfix/pickup dev="vda1" ino=256669 scontext=system_u:system_r:postfix_master_t:s0 tcontext=system_u:object_r:postfix_pickup_exec_t:s0 tclass=file permissive=1 
----
type=PROCTITLE msg=audit(05/23/2018 03:09:34.560:53) : proctitle=qmgr -l -t unix -u 
type=SYSCALL msg=audit(05/23/2018 03:09:34.560:53) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x56142a917720 a1=0x56142a917b20 a2=0x56142a913680 a3=0xffffffff items=0 ppid=1655 pid=1660 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=qmgr exe=/usr/libexec/postfix/qmgr subj=system_u:system_r:postfix_qmgr_t:s0 key=(null) 
type=AVC msg=audit(05/23/2018 03:09:34.560:53) : avc:  denied  { map } for  pid=1660 comm=qmgr path=/usr/libexec/postfix/qmgr dev="vda1" ino=258785 scontext=system_u:system_r:postfix_master_t:s0 tcontext=system_u:object_r:postfix_qmgr_exec_t:s0 tclass=file permissive=1 
----
type=PROCTITLE msg=audit(05/23/2018 03:10:03.239:73) : proctitle=/usr/sbin/unix_chkpwd root chkexpiry 
type=SYSCALL msg=audit(05/23/2018 03:10:03.239:73) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x7f513b2f634d a1=0x7ffe62c16ab0 a2=0x7f513b4f9368 a3=0x2 items=0 ppid=1700 pid=1703 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=unix_chkpwd exe=/usr/sbin/unix_chkpwd subj=system_u:system_r:chkpwd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(05/23/2018 03:10:03.239:73) : avc:  denied  { map } for  pid=1703 comm=unix_chkpwd path=/usr/sbin/unix_chkpwd dev="vda1" ino=6467114 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:chkpwd_exec_t:s0 tclass=file permissive=1 
----

The machine was running in permissive mode, because /boot/grub2/grub.cfg file contains "enforcing=0" on the kernel command line.

Comment 5 Milos Malik 2018-05-23 07:33:01 UTC
The first SELinux denial in comment#4 is already addressed in BZ#1574383.
I believe the last SELinux denial in comment#4 is the cause why you cannot log in back via ssh.

Comment 6 Milos Malik 2018-05-23 14:43:25 UTC
Following SELinux denial did not make it to audit.log:

# dmesg | grep type=1400
[    3.227820] type=1400 audit(1527086219.186:3): avc:  denied  { map } for  pid=835 comm="audispd" path="/usr/sbin/audispd" dev="vda1" ino=6842012 scontext=system_u:system_r:auditd_t:s0 tcontext=system_u:object_r:audisp_exec_t:s0 tclass=file permissive=1
#

but it appeared when the machine booted in permissive mode:

# rpm -qa kernel\* selinux-policy\* | sort
kernel-3.10.0-890.el7.x86_64
kernel-debug-devel-3.10.0-890.el7.x86_64
kernel-headers-3.10.0-890.el7.x86_64
kernel-tools-3.10.0-890.el7.x86_64
kernel-tools-libs-3.10.0-890.el7.x86_64
selinux-policy-3.13.1-193.el7.noarch
selinux-policy-devel-3.13.1-193.el7.noarch
selinux-policy-targeted-3.13.1-193.el7.noarch
#

Comment 7 Walid A. 2018-05-23 16:38:03 UTC
Info for comment 3:

This after upgrading the kernel to 3.10.0-889.el7 and rebooted,
and BEFORE upgrading selinux-policy, selinux-policy-targeted:
-------------------------------------------------------------

# yum list selinux*
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
selinux-policy.noarch                                     3.13.1-192.el7_5.3                             @oso-rhui-rhel-server-releases
selinux-policy-targeted.noarch                            3.13.1-192.el7_5.3                             @oso-rhui-rhel-server-releases
Available Packages
selinux-policy.noarch                                     3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-devel.noarch                               3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-doc.noarch                                 3.13.1-193.el7                                 rhel-7-optional-next          
selinux-policy-minimum.noarch                             3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-mls.noarch                                 3.13.1-193.el7                                 rhel-7-next                   
selinux-policy-sandbox.noarch                             3.13.1-193.el7                                 rhel-7-optional-next          
selinux-policy-targeted.noarch                            3.13.1-193.el7                                 rhel-7-next                   
root@ip-172-31-16-185: ~ # 

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today
<no matches>

# yum list kernel* 
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
kernel.x86_64                                                       3.10.0-862.el7                                      @anaconda/7.5       
kernel.x86_64                                                       3.10.0-888.el7                                      @rhel-7-next        
kernel.x86_64                                                       3.10.0-889.el7                                      @rhel-7-next        
kernel-devel.x86_64                                                 3.10.0-888.el7                                      @rhel-7-next        
kernel-devel.x86_64                                                 3.10.0-889.el7                                      @rhel-7-next        
kernel-headers.x86_64                                               3.10.0-889.el7                                      @rhel-7-next        
kernel-tools.x86_64                                                 3.10.0-889.el7                                      @rhel-7-next        
kernel-tools-libs.x86_64                                            3.10.0-889.el7                                      @rhel-7-next        
Available Packages
kernel-abi-whitelists.noarch                                        3.10.0-889.el7                                      rhel-7-next         
kernel-debug.x86_64                                                 3.10.0-889.el7                                      rhel-7-next         
kernel-debug-devel.x86_64                                           3.10.0-889.el7                                      rhel-7-next         
kernel-doc.noarch                                                   3.10.0-889.el7                                      rhel-7-next         
kernel-tools-libs-devel.x86_64                                      3.10.0-889.el7                                      rhel-7-optional-next

# uname -r
3.10.0-889.el7.x86_64


############ After upgrading selinux-policy and selinux-policy-targeted:

# yum list selinux-policy*
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Installed Packages
selinux-policy.noarch                                               3.13.1-193.el7                                      @rhel-7-next        
selinux-policy-targeted.noarch                                      3.13.1-193.el7                                      @rhel-7-next        
Available Packages
selinux-policy-devel.noarch                                         3.13.1-193.el7                                      rhel-7-next         
selinux-policy-doc.noarch                                           3.13.1-193.el7                                      rhel-7-optional-next
selinux-policy-minimum.noarch                                       3.13.1-193.el7                                      rhel-7-next         
selinux-policy-mls.noarch                                           3.13.1-193.el7                                      rhel-7-next         
selinux-policy-sandbox.noarch                                       3.13.1-193.el7                                      


#### selinux denials AFTER upgrade selinux-policy*

# ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today
----
type=USER_AVC msg=audit(05/23/2018 16:19:38.829:105) : pid=595 uid=dbus auid=unset ses=unset subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  received policyload notice (seqno=2)  exe=/usr/bin/dbus-daemon sauid=dbus hostname=? addr=? terminal=?'

Comment 8 Lukas Vrabec 2018-05-24 12:49:45 UTC
*** Bug 1581858 has been marked as a duplicate of this bug. ***

Comment 9 Jarod Wilson 2018-05-24 17:17:41 UTC
Was bisecting odd network failures last night myself, and found that it was an selinux kernel change in 3.10.0-375.el7 that started triggering problems for me, got pointed this direction.

Comment 12 Lukas Vrabec 2018-06-03 15:48:01 UTC
*** Bug 1583085 has been marked as a duplicate of this bug. ***

Comment 17 errata-xmlrpc 2018-10-30 10:04:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3111