Description of problem: SELinux policy needed an update to resolve https://bugzilla.redhat.com/show_bug.cgi?id=1966842 But after upgrade, it did not work, and we still received Permission Denied error. Checking the permissions: # matchpathcon /run/libvirt/common/system.token /etc/selinux/targeted/contexts/files/file_contexts: invalid context system_u:object_r:virt_common_var_run_t:s0 matchpathcon(/run/libvirt/common/system.token) failed: Invalid argument Now it seems to be caused by an old binary policy that is migrated: 2021-07-06 15:38:57,636 [DEBUG] (migrate_etc) Calling: (['cp', '-a', '-r', '/tmp/mnt.PCsjo///etc/selinux/targeted/policy/policy.31', '/tmp/mnt.5Gxkk///etc/selinux/targeted/policy/policy.31'],) {'close_fds': True, 'stderr': -2} Think we should never copy the binary policy file, but always generate a new one. How reproducible: Upgrade oVirt 4.4.6 to 4.4.7
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
From IRC: (17:07:57) dupondje: either we don't copy that file or we force a recompile (semodule -B)
Its not a big issue to copy the file indeed, but we should trigger a semodule -B afterwards to make sure a new one is generated. Also I don't know if we need to 'touch /.autorelabel' to fix the incorrect labels after the update?
QE did not fully reproduce this issue. Test versions: RHVH: upgrade RHVH from rhvh-4.4.6.2-0.20210615.0+1 to rhvh-4.4.7.2-0.20210705.0+1 RHVM: 4.4.7.6-0.11.el8ev Test steps: 1. Install redhat-virtualization-host-4.4.6-20210615.0.el8_4 2. Login to the host, setup local repos and point to "redhat-virtualization-host-4.4.7-20210705.0.el8_4" 3. Add RHVH to RHVM 4. On RHVM portal, add NFS Storage Domain 5. Create a VM and start it 6. Power off the VM and upgrade RHVH via RHVM 7. After upgrade, start the VM 8. Checking the permissions: # matchpathcon /run/libvirt/common/system.token Test results: 1. When executing the command in “step 8” to check the permissions, there is no error message ~~~~~~ # matchpathcon /run/libvirt/common/system.token /run/libvirt/common/system.token system_u:object_r:virt_var_run_t:s0 ~~~~~~ 2. After RHVH upgrade, the VM can start normally. 3. I checked /var/log/imgbased.log and and found that the old binary policy has been migrated as you mentioned ~~~~~~ 2021-07-07 05:36:50,186 [DEBUG] (migrate_etc) Calling: (['cp', '-a', '-r', '/tmp/mnt.mNRby///etc/selinux/targeted/policy/policy.31', '/tmp/mnt.4xtCX///etc/selinux/targeted/policy/policy.31'],) {'close_fds': True, 'stderr': -2} ~~~~~~ Additional info: I started the VM via the RHVM portal instead of the command line, I didn’t see Permission denied. Could you please give me some detailed steps on how to reproduce this bug? Thank you.
Hey, Well the error could be caused because the policy file is even older (pre 4.4.6). But as you see your matchpathcon /run/libvirt/common/system.token is wrong. You have: matchpathcon /run/libvirt/common/system.token /run/libvirt/common/system.token system_u:object_r:virt_var_run_t:s0 It should be: # matchpathcon /run/libvirt/common/system.token /run/libvirt/common/system.token system_u:object_r:virt_common_var_run_t:s0 See https://github.com/fedora-selinux/selinux-policy/pull/777
@peyu please note this affects oVirt Node (based on CentOS Stream / RHEL 8.5 pre-release) but not RHV-H (RHEL 8.4.0.1) because the critical changes have been introduced in libvirt 7.4. This also should not affect hosts that are not running oVirt Node (CentOS Stream based hosts, other RHEL 8.4 derivatives).
This also should not affect clean oVirt Node 4.4.7 installations, only oVirt Node upgraded from previous versions.
I am a little confused, should I reproduce this issue on the upstream oVirt node? Should I check: 1) clean installations of oVirt Node (based on CentOS Stream / RHEL 8.5 pre-release) 2) upgrade oVirt Node from previous versions to oVirt Node 4.4.7 ? Or, according to Comment 5, does it mean that Comment 4 has reproduced the issue in imgbased?
@Peyu: 1) Clean installs are NOT affected (as it has no old policy file there to copy) 2) Upgrade oVirt Node from previous version (4.4.5/4.4.6) to 4.4.7 should trigger it. If your RHVH has selinux-policy-targeted-3.14.3-71.el8.noarch, then you have reproduced it already indeed. Because selinux-policy-targeted-3.14.3-71.el8.noarch contains a rule that should mark /run/libvirt/common/system.token as virt_common_var_run_t and not virt_var_run_t.
@jean-louis Thank you very much for the information. In Comment 4, RHVH has selinux-policy-targeted-3.14.3-67.el8.noarch. So I will try to reproduce this bug with oVirt node.
Also please note that it reproduces only if oVirt Node from 4.4.5/4.4.6 was attached to the engine before upgrading to 4.4.7. We noticed that if you just install 4.4.6 and upgrade to 4.4.7 without attaching oVirt Node to the engine the bug doesn't reproduce.
QE reproduced this issue with oVirt node. Test versions: RHVH: upgrade oVirt node from ovirt-node-ng-4.4.6.3-0.20210518.0+1 to ovirt-node-ng-4.4.7-0.20210705.0+1 engine: 4.4.4.5-1.el8 Test steps: 1. Install ovirt-node-ng-installer-4.4.6-2021051809.el8.iso 2. Login to the node, setup local repos and point to "ovirt-node-ng-image-update-4.4.7-1.el8.noarch.rpm" 3. Add the node to engine 4. On engine portal, add NFS Storage Domain 5. Create a VM and start it 6. Power off the VM and upgrade the node via engine 7. After upgrade, start the VM 8. Checking the permissions: # matchpathcon /run/libvirt/common/system.token Test results: 1. After upgrade, the VM can not be started, its Status is "Down" 2. The permissions is wrong ~~~~~~ # matchpathcon /run/libvirt/common/system.token /etc/selinux/targeted/contexts/files/file_contexts: invalid context system_u:object_r:virt_common_var_run_t:s0 matchpathcon(/run/libvirt/common/system.token) failed: Invalid argument ~~~~~~ Additional info: I. After upgrade oVirt node ~~~~~~ # imgbase w You are on ovirt-node-ng-4.4.7-0.20210705.0+1 # imgbase layout ovirt-node-ng-4.4.6.3-0.20210518.0 +- ovirt-node-ng-4.4.6.3-0.20210518.0+1 ovirt-node-ng-4.4.7-0.20210705.0 +- ovirt-node-ng-4.4.7-0.20210705.0+1 # rpm -qa | grep libvirt libvirt-daemon-config-nwfilter-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-mpath-7.4.0-1.el8s.x86_64 libvirt-libs-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-nwfilter-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-secret-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-iscsi-direct-7.4.0-1.el8s.x86_64 libvirt-lock-sanlock-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-logical-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-qemu-7.4.0-1.el8s.x86_64 python3-libvirt-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-interface-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-iscsi-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-scsi-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-nodedev-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-network-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-core-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-gluster-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-rbd-7.4.0-1.el8s.x86_64 libvirt-daemon-config-network-7.4.0-1.el8s.x86_64 libvirt-client-7.4.0-1.el8s.x86_64 libvirt-7.4.0-1.el8s.x86_64 libvirt-daemon-kvm-7.4.0-1.el8s.x86_64 libvirt-daemon-7.4.0-1.el8s.x86_64 libvirt-daemon-driver-storage-disk-7.4.0-1.el8s.x86_64 # rpm -qa | grep selinux-policy selinux-policy-3.14.3-71.el8.noarch selinux-policy-targeted-3.14.3-71.el8.noarch ~~~~~~
Test build that fixes this issue and includes 5.2.0 version of qemu packages: https://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el8-x86_64/784/
Seems like the patch causes other issues now: Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: Error: Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: One of the modules is not configured to work with VDSM. Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: To configure the module use the following: Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: 'vdsm-tool configure [--module module-name]'. Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: If all modules are not configured try to use: Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: 'vdsm-tool configure --force' Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: (The force flag will stop the module's service and start it Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: afterwards automatically to load the new configuration.) Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: libvirt is already configured for vdsm Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: sanlock is configured for vdsm Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: Current revision of multipath.conf detected, preserving Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: lvm is configured for vdsm Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: Managed volume database is already configured Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: abrt is already configured for vdsm Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: Modules sebool are not configured Jul 13 11:24:03 ovn002 vdsmd_init_common.sh[7783]: vdsm: stopped during execute check_is_configured task (task returned with error code 1). Can be fixed by a semodule -B and reboot. But right after the upgrade vdsmd doesn't want to start!
Hi Jean-Louis, Can you please test this build? : https://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el8-x86_64/786/artifact/exported-artifacts/ovirt-node-ng-installer-4.4.7-2021071400.el8.iso https://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el8-x86_64/786/artifact/exported-artifacts/ovirt-node-ng-image-update-4.4.7.1-1.el8.noarch.rpm It includes the fix for the VDSM issue you encountered.
(In reply to Lev Veyde from comment #17) > Hi Jean-Louis, > > Can you please test this build? : > > https://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el8- > x86_64/786/artifact/exported-artifacts/ovirt-node-ng-installer-4.4.7- > 2021071400.el8.iso QE checked the above upgrade package and it is correct. After upgrade, the permission is correct, the node can be "up" and the VM can be started. > https://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-el8- > x86_64/786/artifact/exported-artifacts/ovirt-node-ng-image-update-4.4.7.1-1. > el8.noarch.rpm The above installation package is correct. The permission is “/run/libvirt/common/system.token system_u:object_r:virt_common_var_run_t:s0”. Add the node to engine, create a VM, and the VM can be started. > It includes the fix for the VDSM issue you encountered.
Hey Lev, That works like it should :) Thanks
(In reply to Jean-Louis Dupond from comment #19) > Hey Lev, > > That works like it should :) > Thanks Great :-) Thanks for the update (and opening the bug).
QE did not found the upgrade package "ovirt-4.4.7-2" from https://resources.ovirt.org/pub/ovirt-4.4/rpm/el8/noarch/ Pending on new build.
The latest version for node, that includes the fix is 4.4.7.1-1, thanks Lev for the information. QE verified this issue on "ovirt-node-ng-image-update-4.4.7.1-1.el8.noarch.rpm" Test versions: RHVH: upgrade oVirt node from ovirt-node-ng-4.4.6.3-0.20210518.0+1 to ovirt-node-ng-4.4.7.1-0.20210713.0+1 engine: 4.4.4.5-1.el8 Test steps: 1. Install ovirt-node-ng-installer-4.4.6-2021051809.el8.iso 2. Login to the node, setup local repos and point to "ovirt-node-ng-image-update-4.4.7.1-1.el8.noarch.rpm" 3. Add the node to engine 4. On engine portal, add NFS Storage Domain 5. Create a VM and start it 6. Power off the VM and upgrade the node via engine 7. After upgrade, start the VM 8. Checking the permissions: # matchpathcon /run/libvirt/common/system.token Test results: 1. After upgrade, the VM can start, its Status is "Up" 2. The permissions is correct ~~~~~~ # matchpathcon /run/libvirt/common/system.token /run/libvirt/common/system.token system_u:object_r:virt_common_var_run_t:s0 ~~~~~~ Move the bug Status to "VERIFIED".