Bug 1381588

Summary: sebooleans get reset on image based systems / RHVH status is Non Responsive in RHVM side after upgrade from RHVH 4.0_7.2 to 4.0_7.3
Product: Red Hat Enterprise Linux 7 Reporter: Fabian Deutsch <fdeutsch>
Component: selinux-policyAssignee: Lukas Vrabec <lvrabec>
Status: CLOSED ERRATA QA Contact: Milos Malik <mmalik>
Severity: urgent Docs Contact: Mirek Jahoda <mjahoda>
Priority: urgent    
Version: 7.3CC: bugs, cshao, dguo, dougsland, fdeutsch, huzhao, jiawu, jneedle, leiwang, lmiksik, lvrabec, mgoldboi, mgrepl, miabbott, mjahoda, mkolaja, mmalik, msivak, plautrba, pvrabec, rbarry, sherold, snagar, ssekidde, walters, weiwang, yaniwang, ybronhei, ycui, yruseva, yzhao
Target Milestone: rcKeywords: TestBlocker, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
In Red Hat Enterprise Linux 7.3, the SELinux user space uses the different location for some files, compared to the previous versions of Red Hat Enterprise Linux 7. Consequently, Red Hat Virtualization Host (RHVH) or Red Hat Atomic Host (RHAH) had non-responsive status, in some cases. The migrate script to perform the change from the old modules store structure to the new one is now provided.
Story Points: ---
Clone Of: 1373389
: 1383450 (view as bug list) Environment:
Last Closed: 2017-08-01 15:15:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1372376, 1373389, 1375561, 1383450    

Description Fabian Deutsch 2016-10-04 13:34:55 UTC
In RHEL 7.3 the SELinux userspace rebase changed paths.
On systems which don't run %post-lets (like atomic and RHVH) this is causing trouble, because the local polciies are not relocated to the new location.

This has a strong impact on the high-level functionality.

+++ This bug was initially created as a clone of Bug #1373389 +++

…

Diff from getsebool
====================
# diff -ruN before-upgrade after-upgrade 
--- before-upgrade	2016-09-26 23:54:51.603765224 -0400
+++ after-upgrade	2016-09-26 23:54:34.891341349 -0400
@@ -36,6 +36,7 @@
 deny_ptrace --> off
 dhcpc_exec_iptables --> off
 dhcpd_use_ldap --> off
+docker_connect_any --> off
 domain_fd_use --> on
 domain_kernel_load_modules --> off
 entropyd_use_audio --> on
@@ -46,7 +47,6 @@
 fenced_can_network_connect --> off
 fenced_can_ssh --> off
 fips_mode --> on
-ftp_home_dir --> off
 ftpd_anon_write --> off
 ftpd_connect_all_unreserved --> off
 ftpd_connect_db --> off
@@ -129,6 +129,7 @@
 logging_syslogd_run_nagios_plugins --> off
 logging_syslogd_use_tty --> on
 login_console_enabled --> on
+logrotate_read_inside_containers --> off
 logrotate_use_nfs --> off
 logwatch_can_network_connect_mail --> off
 lsmd_plugin_connect_any --> off
@@ -202,9 +203,9 @@
 samba_run_unconfined --> off
 samba_share_fusefs --> off
 samba_share_nfs --> off
-sanlock_use_fusefs --> on
-sanlock_use_nfs --> on
-sanlock_use_samba --> on
+sanlock_use_fusefs --> off
+sanlock_use_nfs --> off
+sanlock_use_samba --> off
 saslauthd_read_shadow --> off
 secadm_exec_content --> on
 secure_mode --> off
@@ -222,16 +223,13 @@
 selinuxuser_tcp_server --> off
 selinuxuser_udp_server --> off
 selinuxuser_use_ssh_chroot --> off
-sftpd_anon_write --> off
-sftpd_enable_homedirs --> off
-sftpd_full_access --> off
-sftpd_write_ssh_home --> off
 sge_domain_can_network_connect --> off
 sge_use_nfs --> off
 smartmon_3ware --> off
 smbd_anon_write --> off
 spamassassin_can_network --> off
 spamd_enable_home_dirs --> on
+spamd_update_can_network --> off
 squid_connect_any --> on
 squid_use_tproxy --> off
 ssh_chroot_rw_homedirs --> off
@@ -245,6 +243,7 @@
 telepathy_tcp_connect_generic_network_ports --> on
 tftp_anon_write --> off
 tftp_home_dir --> off
+tmpreaper_use_cifs --> off
 tmpreaper_use_nfs --> off
 tmpreaper_use_samba --> off
 tor_bind_all_unreserved_ports --> off
@@ -264,19 +263,18 @@
 virt_rw_qemu_ga_data --> off
 virt_sandbox_use_all_caps --> on
 virt_sandbox_use_audit --> on
+virt_sandbox_use_fusefs --> off
 virt_sandbox_use_mknod --> off
 virt_sandbox_use_netlink --> off
-virt_sandbox_use_nfs --> off
-virt_sandbox_use_samba --> off
 virt_sandbox_use_sys_admin --> off
 virt_transition_userdomain --> off
 virt_use_comm --> off
 virt_use_execmem --> off
-virt_use_fusefs --> on
-virt_use_nfs --> on
+virt_use_fusefs --> off
+virt_use_nfs --> off
 virt_use_rawip --> off
-virt_use_samba --> on
-virt_use_sanlock --> on
+virt_use_samba --> off
+virt_use_sanlock --> off
 virt_use_usb --> on
 virt_use_xserver --> off
 webadm_manage_user_files --> off

--- Additional comment from Douglas Schilling Landgraf on 2016-09-27 07:43:37 CEST ---

Worth to mention that if I upgrade the rpms via yum (non squashfs) from vdsm-4.18.11-1.el7ev to vdsm-4.18.13-1.el7ev.x86_64 I didn't see any problem.
The issue seems related to the boot with the new squashfs and updated vdsm.

Tested executed:
# installed RHVH-4.0-20160822.8-RHVH-x86_64-dvd1.iso 
# registered and approved in RHVM
# created a local repo with vdsm-4.18.13-1.el7ev.x86_64
# yum update -y
# reboot
After reboot, host is up.

Comment 4 Fabian Deutsch 2016-10-06 12:32:18 UTC
Steps ot reproduce on any image based system:

1. Use the old image
2. setsebool virt_use_nfs 1
3. Update
4. getsebool -a | grep virt_use_nfs

After 4 virt_use_nfs is back to 0

Comment 5 Micah Abbott 2016-10-06 14:37:17 UTC
This also affects the upgrade of Atomic Host from 7.2.7 to 7.3

Comment 6 Micah Abbott 2016-10-06 14:48:43 UTC
I believe, this comment (and those that follow) linked here in an upstream bug are related - https://bugzilla.redhat.com/show_bug.cgi?id=1290659#c42

Comment 7 Petr Lautrbach 2016-10-06 16:19:11 UTC
It's 2 different issues. In Fedora, the store is completely removed while on RHEL-7.3 are some files in the different location then tools expects them.

These files could be migrated during first boot after update. The question is if this service can be part of Atomic Host similar to /usr/lib/tpmfiles.d/ files or if they have to be shipped by selinux-policy package.

Comment 8 Fabian Deutsch 2016-10-06 21:01:58 UTC
As there are at least two products affected I'd really favor a selinux based solution.

Comment 12 Micah Abbott 2016-10-07 19:23:56 UTC
Yoana, this will need a release note for RHELAH 7.3.  If a fix is provided before we release, we can drop the release note.

Comment 13 Petr Lautrbach 2016-10-10 09:05:27 UTC
(In reply to Fabian Deutsch from comment #4)
> Steps ot reproduce on any image based system:
> 
> 1. Use the old image
> 2. setsebool virt_use_nfs 1
> 3. Update
> 4. getsebool -a | grep virt_use_nfs
> 
> After 4 virt_use_nfs is back to 0

Is there a reboot between setsebool and getsebool? Then you need to use 'setsebool -P virt_use_nfs 1' to make the change persistent.

Comment 24 Lukas Vrabec 2016-10-10 21:03:18 UTC
Ryan, 

Did you add "-P" parameter with setsebool before update?

Comment 25 Ryan Barry 2016-10-10 21:05:38 UTC
I did:

[root@localhost ~]# setsebool -P virt_use_nfs 1
[root@localhost ~]# getsebool -a | grep virt_use_nfs
virt_use_nfs --> on
[root@localhost ~]# ls
anaconda-ks.cfg  redhat-virtualization-host-image-update-4.0-20161010.0.el7_3.noarch.rpm
[root@localhost ~]# rpm -Uvh redhat-virtualization-host-image-update-4.0-20161010.0.el7_3.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:redhat-virtualization-host-image-################################# [ 50%]
Cleaning up / removing...
   2:redhat-virtualization-host-image-################################# [100%]
[root@localhost ~]# reboot
PolicyKit daemon disconnected from the bus.
We are no longer a registered authentication agent.
Terminated
[root@localhost ~]# Connection to 192.168.122.65 closed by remote host.
Connection to 192.168.122.65 closed.
[rbarry@el7 redhat-virtualization-host]$ ssh root.122.65
root.122.65's password: 
Last login: Mon Oct 10 08:55:37 2016 from el7.nested

  imgbase status: OK

[root@localhost ~]# !get^C
[root@localhost ~]# getsebool -a | grep virt_use
virt_use_comm --> off
virt_use_execmem --> off
virt_use_fusefs --> off
virt_use_nfs --> off
virt_use_rawip --> off
virt_use_samba --> off
virt_use_sanlock --> off
virt_use_usb --> on
virt_use_xserver --> off

Comment 26 Ryan Barry 2016-10-10 21:08:40 UTC
Note that RHV-H (similar to Atomic in this sense) is updated in an A/B format.

The RPM contains a squashfs, which is delivered onto a new LVM LV. After this, /etc and /root are synced (and UID/GID/permissions drift which may have happened between the two images is corrected).

Because of this, any %post scripts from selinux-policy (as an example) will not be triggered on the actual update. 

For RHV-H (and Atomic, presumably), the migration of changes must occur AFTER the reboot, not as part of selinux-policy-targeted %post or similar. %post will be run in that case on a brand-new system which doesn't have any policy changes to migrate...

Comment 27 Lukas Vrabec 2016-10-11 08:44:21 UTC
Ryan, 
Understand, but we have patch which fixes this after reboot by using systemd-unit file. Patch is attached in comment#16. Could you please prepare RHV-H host without update for testing purposes? I'll try to reproduce it. 

Thanks.

Comment 28 Petr Lautrbach 2016-10-11 08:49:14 UTC
Please provide an output of the following commands after update:

# systemctl status -l selinux-policy-migrate-local-changes

# ls -l /etc/selinux/targeted/modules/active/

# cat /etc/selinux/targeted/modules/active/booleans.local

Comment 30 Ryan Barry 2016-10-11 13:50:19 UTC
[root@localhost ~]# systemctl status -l selinux-policy-migrate-local-changes
● selinux-policy-migrate-local-changes - Migrate local SELinux policy changes from the old store structure to the new structure
   Loaded: loaded (/usr/lib/systemd/system/basic.target.wants/../selinux-policy-migrate-local-changes@.service; static; vendor preset: disabled)
   Active: active (exited) since Mon 2016-10-10 10:34:30 MST; 20h ago
  Process: 794 ExecStart=/usr/libexec/selinux/selinux-policy-migrate-local-changes.sh %I (code=exited, status=0/SUCCESS)
 Main PID: 794 (code=exited, status=0/SUCCESS)

Oct 10 10:34:20 localhost.localdomain systemd[1]: Starting Migrate local SELinux policy changes from the old store structure to the new structure...
Oct 10 10:34:30 localhost.localdomain systemd[1]: Started Migrate local SELinux policy changes from the old store structure to the new structure.
[root@localhost ~]# ls -l /etc/selinux/targeted/modules/active/
total 864
-rw-r--r--. 1 root root  58507 Oct 10 10:17 base.pp
-rw-r--r--. 1 root root     85 Oct 10 10:17 booleans.local
-rw-------. 1 root root     32 Oct 10 10:17 commit_num
-rw-------. 1 root root 368557 Oct 10 10:17 file_contexts
-rw-r--r--. 1 root root  13169 Oct 10 10:17 file_contexts.homedirs
-rw-r--r--. 1 root root    127 Oct 10 10:17 file_contexts.local
-rw-------. 1 root root 380333 Oct 10 10:17 file_contexts.template
-rw-------. 1 root root  11776 Oct 10 10:17 homedir_template
drwx------. 2 root root  12288 Oct 10 10:17 modules
-rw-------. 1 root root      0 Oct 10 10:17 netfilter_contexts
lrwxrwxrwx. 1 root root     38 Oct 10 10:17 policy.kern -> /etc/selinux/targeted/policy/policy.29
-rw-r--r--. 1 root root    136 Oct 10 10:17 ports.local
-rw-r--r--. 1 root root    282 Oct 10 10:34 README.migrated
-rw-------. 1 root root    106 Oct 10 10:17 seusers.final
-rw-------. 1 root root    101 Oct 10 10:17 users_extra
[root@localhost ~]# cat /etc/selinux/targeted/modules/active/booleans.local 
# This file is auto-generated by libsemanage
# Do not edit directly.

virt_use_nfs=1
[root@localhost ~]# getsebool -a | grep virt_use_nfs
virt_use_nfs --> off

Comment 31 Ryan Barry 2016-10-11 13:51:35 UTC
(In reply to Lukas Vrabec from comment #27)
> Ryan, 
> Understand, but we have patch which fixes this after reboot by using
> systemd-unit file. Patch is attached in comment#16. Could you please prepare
> RHV-H host without update for testing purposes? I'll try to reproduce it. 
> 
> Thanks.

I have a libvirt snapshot before the update, so testing should be easy.

Are you asking for a publicly available test system here? I'm not sure of the request.

Comment 33 Petr Lautrbach 2016-10-11 14:50:36 UTC
Since there's /etc/selinux/targeted/modules/active/README.migrated, the migration happened. It's not clear to me why local changes are not copied to the new store.

Can you please run:

# rm -f /etc/selinux/targeted/modules/active/README.migrated
# bash -x /usr/libexec/selinux/selinux-policy-migrate-local-changes.sh targeted
# cat /etc/selinux/targeted/modules/active/booleans.local 
# getsebool -a | grep virt_use_nfs
# cat /etc/selinux/targeted/active/booleans.local


Or provide an access to some machine or the reproducer?

Comment 40 Petr Lautrbach 2016-10-11 18:17:17 UTC
The problem seems to be in the migration scripts which migrates changes but doesn't apply them:

# semanage boolean -C -l
SELinux boolean                State  Default Description

cups_execmem                   (off  ,   on)  Allow cups to execmem
virt_use_nfs                   (off  ,   on)  Allow virt to use nfs

The system default is 'on' which is correct since it was changed before update, but the system runs with 'off' as the policy which is loaded was build with 'off' and the local changes were not applied.


I'll provide another build with fixed script.

Comment 41 Petr Lautrbach 2016-10-12 06:52:01 UTC
According to my testing, it's fixed in https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=11892056  Please confirm.

All changes are migrated and applied in running system.

Ryan, thanks for the help with setting an environment.

Comment 42 Ryan Barry 2016-10-12 15:58:32 UTC
Confirmed, this works.

Thanks!

Comment 46 errata-xmlrpc 2017-08-01 15:15:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1861