Bug 1394426

Summary: After minor update instances will not start
Product: Red Hat OpenStack Reporter: Randy Perryman <randy_perryman>
Component: openstack-selinuxAssignee: Ryan Hallisey <rhallise>
Status: CLOSED CURRENTRELEASE QA Contact: Udi Shkalim <ushkalim>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: arkady_kanevsky, awaugama, berrange, cdevine, christopher_dearborn, dasmith, dcain, ebarrera, eglynn, John_walsh, kasmith, kchamart, kurt_hey, lhh, mgrepl, morazi, nlevinki, ochalups, randy_perryman, sbauza, sferdjao, sgordon, smerrow, sreichar, srevivo, vromanso
Target Milestone: ---   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1396393 (view as bug list) Environment:
Last Closed: 2016-11-14 21:30:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1305654, 1396393    
Attachments:
Description Flags
SOSreport from one of the computes. none

Description Randy Perryman 2016-11-11 23:22:39 UTC
Completed a minor update and now instances will not start.  The following error is in the logs:

016-11-11 23:14:30.153 26795 ERROR nova.scheduler.utils [req-0f2cf531-1790-44c3-841d-d7e054af9cc3 4aa6acb4fa6d462da14e632dd367ae06 c3d7642b2cac4391aa1b50d075913f6b - - -] [instance: 8a8d696a-0756-4cc4-8303-28bc2e4faf2e] Error from last host: overcloud-compute-0.localdomain (node overcloud-compute-0.localdomain): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance\n    filter_properties)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2082, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 8a8d696a-0756-4cc4-8303-28bc2e4faf2e was re-scheduled: Unable to open file: /var/lib/nova/instances/8a8d696a-0756-4cc4-8303-28bc2e4faf2e/console.log: Permission denied\n']



Updated OSP 8 to latest using openstack overcloud update...  command and rebooted all nodes successufully

Comment 1 Randy Perryman 2016-11-11 23:26:56 UTC
after the install the dir permission look like this:

heat-admin@overcloud-compute-2 nova]$ cd instances/
[heat-admin@overcloud-compute-2 instances]$ ls -alR
.:
total 4
drwxr-xr-x. 5 nova nova 93 Nov 11 23:14 .
drwxr-xr-x. 8 nova nova 81 Aug 18 17:07 ..
drwxr-xr-x. 2 nova nova 42 Nov 11 19:02 63d01c3c-2903-45a7-bbf8-1610deb42758
drwxr-xr-x. 2 nova nova 53 Nov 11 23:14 _base
-rw-r--r--. 1 nova nova 53 Nov 11 23:00 compute_nodes
drwxr-xr-x. 2 nova nova 91 Nov 11 23:14 locks

./63d01c3c-2903-45a7-bbf8-1610deb42758:
total 4
drwxr-xr-x. 2 nova nova   42 Nov 11 19:02 .
drwxr-xr-x. 5 nova nova   93 Nov 11 23:14 ..
-rw-r--r--. 1 root root    0 Nov 11 19:02 console.log
-rw-r--r--. 1 nova nova 3525 Nov 11 19:02 libvirt.xml

./_base:
total 17748
drwxr-xr-x. 2 nova nova       53 Nov 11 23:14 .
drwxr-xr-x. 5 nova nova       93 Nov 11 23:14 ..
-rw-r--r--. 1 nova nova 41126400 Nov 11 23:14 8810ebd127c19eb15b286d1630765352dca93b03

./locks:
total 0
drwxr-xr-x. 2 nova nova 91 Nov 11 23:14 .
drwxr-xr-x. 5 nova nova 93 Nov 11 23:14 ..
-rw-r--r--. 1 nova nova  0 Nov 11 23:14 nova-8810ebd127c19eb15b286d1630765352dca93b03
-rw-r--r--. 1 nova nova  0 Nov 11 18:51 nova-storage-registry-lock

Comment 2 Randy Perryman 2016-11-11 23:27:22 UTC
BEFORE the Update
.:
total 4
drwxr-xr-x. 4 nova nova 91 Nov 11 19:04 .
-rw-r--r--. 1 root root  0 Nov 11 19:04 foo
drwxr-xr-x. 2 nova nova 42 Nov 11 19:02 63d01c3c-2903-45a7-bbf8-1610deb42758
-rw-r--r--. 1 nova nova 54 Nov 11 18:51 compute_nodes
drwxr-xr-x. 2 nova nova 39 Nov 11 18:51 locks
drwxr-xr-x. 8 nova nova 81 Apr 15  2016 ..

./63d01c3c-2903-45a7-bbf8-1610deb42758:
total 4
drwxr-xr-x. 4 nova nova   91 Nov 11 19:04 ..
drwxr-xr-x. 2 nova nova   42 Nov 11 19:02 .
-rw-r--r--. 1 nova nova 3525 Nov 11 19:02 libvirt.xml
-rw-r--r--. 1 qemu qemu    0 Nov 11 19:02 console.log

./locks:
total 0
drwxr-xr-x. 4 nova nova 91 Nov 11 19:04 ..
drwxr-xr-x. 2 nova nova 39 Nov 11 18:51 .
-rw-r--r--. 1 nova nova  0 Nov 11 18:51 nova-storage-registry-lock

Comment 3 arkady kanevsky 2016-11-11 23:38:38 UTC
Randy,
does the same try for JS-6.0, OSP9 also?

Comment 4 Randy Perryman 2016-11-12 20:19:53 UTC
We have not had that problem installing OSP 9(Mitaka), and in the past (Prior to recent CDN Updates) this test was passing.


This is a Regression.

Comment 5 Randy Perryman 2016-11-12 20:26:13 UTC
Additional information:

If I try to start an exisiting VM with virsh:

Last login: Sat Nov 12 20:23:34 2016 from gateway
[heat-admin@overcloud-compute-2 ~]$ sudo -i
[root@overcloud-compute-2 ~]# virsh start instance-00000003
error: Failed to start domain instance-00000003
error: Unable to open file: /var/lib/nova/instances/63d01c3c-2903-45a7-bbf8-1610deb42758/console.log: Permission denied

[root@overcloud-compute-2 ~]#

Comment 6 Stephen Gordon 2016-11-13 20:30:42 UTC
Lon is this potentially related to required SELinux updates for 7.3? Randy can you confirm you are using RHEL 7.3 here?

Thanks,

Steve

Comment 7 arkady kanevsky 2016-11-14 02:10:51 UTC
Steve,
yes, minor update pulls RHEL-7.3

Comment 8 Randy Perryman 2016-11-14 14:01:21 UTC
Yes, RHEL Version is now 7.3.

Comment 9 Randy Perryman 2016-11-14 14:41:08 UTC
Created attachment 1220463 [details]
SOSreport from one of the computes.

Comment 10 Randy Perryman 2016-11-14 19:42:31 UTC
Setting SELINUX to Permissive allows for VM's to be created and old ones booted.

What is the fix to put SELINUX into enforcing?

Comment 11 Lon Hohberger 2016-11-14 21:30:45 UTC
#============= virtlogd_t ==============

#!!!! This avc is allowed in the current policy
allow virtlogd_t nova_var_lib_t:dir search;


Try openstack-selinux 0.7.11, available in the OSP8 channel.

Comment 12 Randy Perryman 2016-11-14 23:13:33 UTC
Validated latest yum has selinux 0.7.11 and enforcing on the computes now works.