Hide Forgot
Description of problem: Cannot create instances on RHOSP 9 after doing an allinone packstack installation on RHEL7.3. Version-Release number of selected component (if applicable): RHOSP 9 RHEL 7.3 How reproducible: Always Steps to Reproduce: 1. Install RHEL7.3 server 2. packstack --allinone 3. Try to deploy instance Actual results: Instance fails to spawn with error: 2016-11-09 12:12:59.558 3927 ERROR nova.scheduler.utils [req-fbd066e5-a9e4-4c6d-a5a9-7691389992c7 63dcf2074f6b4b7caebd19a8f8228c2d 0fb168d41bc5499d8bbfbb22fac64863 - - -] [instance: e48ea5f8-ffe6-4d94-bb04-0b05072da88c] Error from last host: openstack.oldstables (node openstack.oldstables): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1926, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2116, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance e48ea5f8-ffe6-4d94-bb04-0b05072da88c was re-scheduled: Unable to open file: /var/lib/nova/instances/e48ea5f8-ffe6-4d94-bb04-0b05072da88c/console.log: Permission denied\n'] 2016-11-09 12:12:59.591 3927 WARNING nova.scheduler.utils [req-fbd066e5-a9e4-4c6d-a5a9-7691389992c7 63dcf2074f6b4b7caebd19a8f8228c2d 0fb168d41bc5499d8bbfbb22fac64863 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available. Expected results: Instance should create correctly. Additional info: sealert output: SELinux is preventing /usr/sbin/virtlogd from search access on the directory nova. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that virtlogd should be allowed search access on the nova directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd # semodule -i my-virtlogd.pp Additional Information: Source Context system_u:system_r:virtlogd_t:s0-s0:c0.c1023 Target Context system_u:object_r:nova_var_lib_t:s0 Target Objects nova [ dir ] Source virtlogd Source Path /usr/sbin/virtlogd Port <Unknown> Host <Unknown> Source RPM Packages libvirt-daemon-2.0.0-10.el7.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-102.el7_3.4.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name openstack.oldstables Platform Linux openstack.oldstables 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 Alert Count 4 First Seen 2016-11-09 11:53:49 UTC Last Seen 2016-11-09 16:03:41 UTC Local ID 192b1207-d9bf-4e56-954f-512ca59d1f1d Raw Audit Messages type=AVC msg=audit(1478707421.526:4058): avc: denied { search } for pid=14008 comm="virtlogd" name="nova" dev="dm-0" ino=68052278 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=dir type=SYSCALL msg=audit(1478707421.526:4058): arch=x86_64 syscall=open success=no exit=EACCES a0=7f6ee4000bf0 a1=80441 a2=180 a3=7f6ee4000940 items=0 ppid=1 pid=14008 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) Hash: virtlogd,virtlogd_t,nova_var_lib_t,dir,search ------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/virtlogd from getattr access on the file /var/lib/nova/instances/e1dbd2ed-4a35-4557-8e30-3e35e7eda7af/console.log. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that virtlogd should be allowed getattr access on the console.log file by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd # semodule -i my-virtlogd.pp Additional Information: Source Context system_u:system_r:virtlogd_t:s0-s0:c0.c1023 Target Context system_u:object_r:nova_var_lib_t:s0 Target Objects /var/lib/nova/instances/e1dbd2ed- 4a35-4557-8e30-3e35e7eda7af/console.log [ file ] Source virtlogd Source Path /usr/sbin/virtlogd Port <Unknown> Host <Unknown> Source RPM Packages libvirt-daemon-2.0.0-10.el7.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-102.el7_3.4.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name openstack.oldstables Platform Linux openstack.oldstables 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 Alert Count 2 First Seen 2016-11-09 12:58:15 UTC Last Seen 2016-11-09 13:57:27 UTC Local ID 304b5593-d8bf-4f74-bed3-2e82f521f791 Raw Audit Messages type=AVC msg=audit(1478699847.49:2426): avc: denied { getattr } for pid=14008 comm="virtlogd" path="/var/lib/nova/instances/e1dbd2ed-4a35-4557-8e30-3e35e7eda7af/console.log" dev="dm-0" ino=33618787 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file type=SYSCALL msg=audit(1478699847.49:2426): arch=x86_64 syscall=fstat success=yes exit=0 a0=10 a1=7f6ee8e218f0 a2=7f6ee8e218f0 a3=7f6ee4000d30 items=0 ppid=1 pid=14008 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) Hash: virtlogd,virtlogd_t,nova_var_lib_t,file,getattr -------------------------------------------------------------------------------- SELinux is preventing /usr/sbin/virtlogd from using the dac_override capability. ***** Plugin dac_override (91.4 confidence) suggests ********************** If you want to help identify if domain needs this access or you have a file with the wrong permissions on your system Then turn on full auditing to get path information about the offending file and generate the error again. Do Turn on full auditing # auditctl -w /etc/shadow -p w Try to recreate AVC. Then execute # ausearch -m avc -ts recent If you see PATH record check ownership/permissions on file, and fix it, otherwise report as a bugzilla. ***** Plugin catchall (9.59 confidence) suggests ************************** If you believe that virtlogd should have the dac_override capability by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'virtlogd' --raw | audit2allow -M my-virtlogd # semodule -i my-virtlogd.pp Additional Information: Source Context system_u:system_r:virtlogd_t:s0-s0:c0.c1023 Target Context system_u:system_r:virtlogd_t:s0-s0:c0.c1023 Target Objects Unknown [ capability ] Source virtlogd Source Path /usr/sbin/virtlogd Port <Unknown> Host <Unknown> Source RPM Packages libvirt-daemon-2.0.0-10.el7.x86_64 Target RPM Packages Policy RPM selinux-policy-3.13.1-102.el7_3.4.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name openstack.oldstables Platform Linux openstack.oldstables 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 Alert Count 2 First Seen 2016-11-09 12:58:15 UTC Last Seen 2016-11-09 13:57:27 UTC Local ID 8d45db24-b5c5-4ecd-916d-49c9068fe6a4 Raw Audit Messages type=AVC msg=audit(1478699847.49:2425): avc: denied { dac_override } for pid=14008 comm="virtlogd" capability=1 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tclass=capability type=AVC msg=audit(1478699847.49:2425): avc: denied { append } for pid=14008 comm="virtlogd" name="console.log" dev="dm-0" ino=33618787 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file type=AVC msg=audit(1478699847.49:2425): avc: denied { open } for pid=14008 comm="virtlogd" path="/var/lib/nova/instances/e1dbd2ed-4a35-4557-8e30-3e35e7eda7af/console.log" dev="dm-0" ino=33618787 scontext=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:nova_var_lib_t:s0 tclass=file type=SYSCALL msg=audit(1478699847.49:2425): arch=x86_64 syscall=open success=yes exit=EBUSY a0=7f6ee4000cd0 a1=80441 a2=180 a3=7f6ee4000d30 items=0 ppid=1 pid=14008 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=virtlogd exe=/usr/sbin/virtlogd subj=system_u:system_r:virtlogd_t:s0-s0:c0.c1023 key=(null) Hash: virtlogd,virtlogd_t,virtlogd_t,capability,dac_override
Just hit this in a lab environment where I only upgraded the following packages (trying to figure out the minimum for a kernel upgrade on compute nodes) dracut dracut-config-generic dracut-config-rescue dracut-network iproute ipxe-roms-qemu kernel kernel-devel kernel-headers kmod libcacard libgudev1 libusbx libvirt libvirt-client libvirt-daemon libvirt-daemon-config-network libvirt-daemon-config-nwfilter libvirt-daemon-driver-interface libvirt-daemon-driver-lxc libvirt-daemon-driver-network libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter libvirt-daemon-driver-qemu libvirt-daemon-driver-secret libvirt-daemon-driver-storage libvirt-daemon-kvm linux-firmware openvswitch python-openvswitch qemu-img-rhev qemu-kvm-common-rhev qemu-kvm-rhev seavgabios-bin seavgabios-bin systemd systemd-libs systemd-sysv usbredir xfsprogs Same thing, using packstack.
The virtlogd problem in openstack-selinux is resolved in all versions. openstack-selinux must be included in the list for minimum packages if you're updating libvirt to 7.3.