Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1847108

Summary: OSP 13 -> 16.1 FFWD upgrade - SELinux permission problem on NFS backed Glance
Product: Red Hat OpenStack Reporter: jritenou
Component: openstack-tripleo-heat-templatesAssignee: Giulio Fidente <gfidente>
Status: CLOSED DUPLICATE QA Contact: David Rosenfeld <drosenfe>
Severity: high Docs Contact:
Priority: urgent    
Version: 16.1 (Train)CC: akekane, athomas, eglynn, ekuvaja, gcharot, gfidente, jpichon, lbezdick, ltoscano, mbracho, mburns, morazi, pgrist, spower
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-03 12:15:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1853433    
Bug Blocks:    
Attachments:
Description Flags
Glance API log from my test environment. none

Description jritenou 2020-06-15 16:51:08 UTC
Created attachment 1697488 [details]
Glance API log from my test environment.

Description of problem: NFS backed glance stores provisioned with OSP 13 do not allow read/write after fast forward upgrade to 16.1 due to SELinux permissions issues.  Have to set enforcement to permissive on controllers to allow read/write.  The controllers themselves mount the glance store fine - it is the glance containers themsleves that are failing to access the store.  


Version-Release number of selected component (if applicable): 16.1


How reproducible: Always - have obvserved this in two environents at least.


Steps to Reproduce:
1. Provision OSP 13 env using director, back glance with NFS and set the necessary selinux context on the store
2. Fast forward upgrade to 16.1
3. Try to upload new image to glance or create instance from existing image

Actual results: Glance cannot read/write to the NFS share


Expected results: Glance should continue working as expected.  


Additional info:

Comment 2 Luigi Toscano 2020-06-17 12:34:14 UTC
Can you please share the glance-specific option passed when deploying 13 (which I guess have been reused verbatim when upgrading on 16.1)?

Can you please share the full logs from all controllers (maybe through sosreport)?

Comment 3 Julie Pichon 2020-06-17 12:41:47 UTC
I am not sure if this is a similar problem, but maybe this bug and THT patches for Nova provide hints on how to resolve: https://bugs.launchpad.net/tripleo/+bug/1869020

If everything works fine once in permissive, could you please include the permissive audit.log? Thank you.

Comment 10 Giulio Fidente 2020-06-23 13:14:59 UTC
(In reply to Julie Pichon from comment #3)
> I am not sure if this is a similar problem, but maybe this bug and THT
> patches for Nova provide hints on how to resolve:
> https://bugs.launchpad.net/tripleo/+bug/1869020
> 
> If everything works fine once in permissive, could you please include the
> permissive audit.log? Thank you.

thanks Julie for bringing this up; I am looking into the upstream bug and issue which seems to be triggered by attempt to relabel the nfs mountpoint due to :z for /var/lib/nova

we don't use :z for /var/lib/glance [1] so there shouldn't be any relabel happening but I am trying to understand if the changes coming with [2] which change the default mount context from 13 to 15 (and 16) are the cause of the denial

1. https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/glance/glance-api-container-puppet.yaml#L617
2. https://review.opendev.org/#/c/682768/

Comment 11 jritenou 2020-06-23 14:13:40 UTC
Hello,

To follow up, I do not have access to the environment I tested with anymore, however, I think I have found the issue in comparing notes with others.  

In 13, I used this line in my storage config file:

GlanceNfsOptions: 'intr,context=system_u:object_r:glance_var_lib_t:s0'

With 16, it appears the selinux context has changed.

GlanceNfsOptions: rw,sync,context=system_u:object_r:container_file_t:s0

Comment 12 Giulio Fidente 2020-06-23 17:43:39 UTC
(In reply to jritenou from comment #11)
> Hello,
> 
> To follow up, I do not have access to the environment I tested with anymore,
> however, I think I have found the issue in comparing notes with others.  
> 
> In 13, I used this line in my storage config file:
> 
> GlanceNfsOptions: 'intr,context=system_u:object_r:glance_var_lib_t:s0'
> 
> With 16, it appears the selinux context has changed.
> 
> GlanceNfsOptions: rw,sync,context=system_u:object_r:container_file_t:s0

note that this is intended; the change comes to fix bz #1742991; we're now assessing on a new environment if on upgrade the fstab line is updated accordingly and if the share is actually remounted with the correct options

Comment 13 Giulio Fidente 2020-06-23 18:47:24 UTC
(In reply to Giulio Fidente from comment #12)
> (In reply to jritenou from comment #11)
> > Hello,
> > 
> > To follow up, I do not have access to the environment I tested with anymore,
> > however, I think I have found the issue in comparing notes with others.  
> > 
> > In 13, I used this line in my storage config file:
> > 
> > GlanceNfsOptions: 'intr,context=system_u:object_r:glance_var_lib_t:s0'
> > 
> > With 16, it appears the selinux context has changed.
> > 
> > GlanceNfsOptions: rw,sync,context=system_u:object_r:container_file_t:s0
> 
> note that this is intended; the change comes to fix bz #1742991; we're now
> assessing on a new environment if on upgrade the fstab line is updated
> accordingly and if the share is actually remounted with the correct options

it looks like the ansible tasks we have in host_prep_tasks [1] which are supposed to refresh fstab *and* remount the share on the host (as per ansible module [2]) are actually only updating the fstab content, not remounting the share because ansible tries to do "-o remount" when possible [3] which isn't sufficient to change the NFS mount options

1. https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/glance/glance-api-container-puppet.yaml#L703
2. https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/modules/system/mount.py#L723
3. https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/modules/system/mount.py#L417

Comment 20 Giulio Fidente 2020-07-03 12:15:06 UTC

*** This bug has been marked as a duplicate of bug 1853433 ***