Hide Forgot
Completed a minor update and now instances will not start. The following error is in the logs: 016-11-11 23:14:30.153 26795 ERROR nova.scheduler.utils [req-0f2cf531-1790-44c3-841d-d7e054af9cc3 4aa6acb4fa6d462da14e632dd367ae06 c3d7642b2cac4391aa1b50d075913f6b - - -] [instance: 8a8d696a-0756-4cc4-8303-28bc2e4faf2e] Error from last host: overcloud-compute-0.localdomain (node overcloud-compute-0.localdomain): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2082, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 8a8d696a-0756-4cc4-8303-28bc2e4faf2e was re-scheduled: Unable to open file: /var/lib/nova/instances/8a8d696a-0756-4cc4-8303-28bc2e4faf2e/console.log: Permission denied\n'] Updated OSP 8 to latest using openstack overcloud update... command and rebooted all nodes successufully
after the install the dir permission look like this: heat-admin@overcloud-compute-2 nova]$ cd instances/ [heat-admin@overcloud-compute-2 instances]$ ls -alR .: total 4 drwxr-xr-x. 5 nova nova 93 Nov 11 23:14 . drwxr-xr-x. 8 nova nova 81 Aug 18 17:07 .. drwxr-xr-x. 2 nova nova 42 Nov 11 19:02 63d01c3c-2903-45a7-bbf8-1610deb42758 drwxr-xr-x. 2 nova nova 53 Nov 11 23:14 _base -rw-r--r--. 1 nova nova 53 Nov 11 23:00 compute_nodes drwxr-xr-x. 2 nova nova 91 Nov 11 23:14 locks ./63d01c3c-2903-45a7-bbf8-1610deb42758: total 4 drwxr-xr-x. 2 nova nova 42 Nov 11 19:02 . drwxr-xr-x. 5 nova nova 93 Nov 11 23:14 .. -rw-r--r--. 1 root root 0 Nov 11 19:02 console.log -rw-r--r--. 1 nova nova 3525 Nov 11 19:02 libvirt.xml ./_base: total 17748 drwxr-xr-x. 2 nova nova 53 Nov 11 23:14 . drwxr-xr-x. 5 nova nova 93 Nov 11 23:14 .. -rw-r--r--. 1 nova nova 41126400 Nov 11 23:14 8810ebd127c19eb15b286d1630765352dca93b03 ./locks: total 0 drwxr-xr-x. 2 nova nova 91 Nov 11 23:14 . drwxr-xr-x. 5 nova nova 93 Nov 11 23:14 .. -rw-r--r--. 1 nova nova 0 Nov 11 23:14 nova-8810ebd127c19eb15b286d1630765352dca93b03 -rw-r--r--. 1 nova nova 0 Nov 11 18:51 nova-storage-registry-lock
BEFORE the Update .: total 4 drwxr-xr-x. 4 nova nova 91 Nov 11 19:04 . -rw-r--r--. 1 root root 0 Nov 11 19:04 foo drwxr-xr-x. 2 nova nova 42 Nov 11 19:02 63d01c3c-2903-45a7-bbf8-1610deb42758 -rw-r--r--. 1 nova nova 54 Nov 11 18:51 compute_nodes drwxr-xr-x. 2 nova nova 39 Nov 11 18:51 locks drwxr-xr-x. 8 nova nova 81 Apr 15 2016 .. ./63d01c3c-2903-45a7-bbf8-1610deb42758: total 4 drwxr-xr-x. 4 nova nova 91 Nov 11 19:04 .. drwxr-xr-x. 2 nova nova 42 Nov 11 19:02 . -rw-r--r--. 1 nova nova 3525 Nov 11 19:02 libvirt.xml -rw-r--r--. 1 qemu qemu 0 Nov 11 19:02 console.log ./locks: total 0 drwxr-xr-x. 4 nova nova 91 Nov 11 19:04 .. drwxr-xr-x. 2 nova nova 39 Nov 11 18:51 . -rw-r--r--. 1 nova nova 0 Nov 11 18:51 nova-storage-registry-lock
Randy, does the same try for JS-6.0, OSP9 also?
We have not had that problem installing OSP 9(Mitaka), and in the past (Prior to recent CDN Updates) this test was passing. This is a Regression.
Additional information: If I try to start an exisiting VM with virsh: Last login: Sat Nov 12 20:23:34 2016 from gateway [heat-admin@overcloud-compute-2 ~]$ sudo -i [root@overcloud-compute-2 ~]# virsh start instance-00000003 error: Failed to start domain instance-00000003 error: Unable to open file: /var/lib/nova/instances/63d01c3c-2903-45a7-bbf8-1610deb42758/console.log: Permission denied [root@overcloud-compute-2 ~]#
Lon is this potentially related to required SELinux updates for 7.3? Randy can you confirm you are using RHEL 7.3 here? Thanks, Steve
Steve, yes, minor update pulls RHEL-7.3
Yes, RHEL Version is now 7.3.
Created attachment 1220463 [details] SOSreport from one of the computes.
Setting SELINUX to Permissive allows for VM's to be created and old ones booted. What is the fix to put SELINUX into enforcing?
#============= virtlogd_t ============== #!!!! This avc is allowed in the current policy allow virtlogd_t nova_var_lib_t:dir search; Try openstack-selinux 0.7.11, available in the OSP8 channel.
Validated latest yum has selinux 0.7.11 and enforcing on the computes now works.