Bug 1847108
| Summary: | OSP 13 -> 16.1 FFWD upgrade - SELinux permission problem on NFS backed Glance | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | jritenou | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | Giulio Fidente <gfidente> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | David Rosenfeld <drosenfe> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 16.1 (Train) | CC: | akekane, athomas, eglynn, ekuvaja, gcharot, gfidente, jpichon, lbezdick, ltoscano, mbracho, mburns, morazi, pgrist, spower | ||||
| Target Milestone: | --- | Keywords: | Triaged | ||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-07-03 12:15:06 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1853433 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
Can you please share the glance-specific option passed when deploying 13 (which I guess have been reused verbatim when upgrading on 16.1)? Can you please share the full logs from all controllers (maybe through sosreport)? I am not sure if this is a similar problem, but maybe this bug and THT patches for Nova provide hints on how to resolve: https://bugs.launchpad.net/tripleo/+bug/1869020 If everything works fine once in permissive, could you please include the permissive audit.log? Thank you. (In reply to Julie Pichon from comment #3) > I am not sure if this is a similar problem, but maybe this bug and THT > patches for Nova provide hints on how to resolve: > https://bugs.launchpad.net/tripleo/+bug/1869020 > > If everything works fine once in permissive, could you please include the > permissive audit.log? Thank you. thanks Julie for bringing this up; I am looking into the upstream bug and issue which seems to be triggered by attempt to relabel the nfs mountpoint due to :z for /var/lib/nova we don't use :z for /var/lib/glance [1] so there shouldn't be any relabel happening but I am trying to understand if the changes coming with [2] which change the default mount context from 13 to 15 (and 16) are the cause of the denial 1. https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/glance/glance-api-container-puppet.yaml#L617 2. https://review.opendev.org/#/c/682768/ Hello, To follow up, I do not have access to the environment I tested with anymore, however, I think I have found the issue in comparing notes with others. In 13, I used this line in my storage config file: GlanceNfsOptions: 'intr,context=system_u:object_r:glance_var_lib_t:s0' With 16, it appears the selinux context has changed. GlanceNfsOptions: rw,sync,context=system_u:object_r:container_file_t:s0 (In reply to jritenou from comment #11) > Hello, > > To follow up, I do not have access to the environment I tested with anymore, > however, I think I have found the issue in comparing notes with others. > > In 13, I used this line in my storage config file: > > GlanceNfsOptions: 'intr,context=system_u:object_r:glance_var_lib_t:s0' > > With 16, it appears the selinux context has changed. > > GlanceNfsOptions: rw,sync,context=system_u:object_r:container_file_t:s0 note that this is intended; the change comes to fix bz #1742991; we're now assessing on a new environment if on upgrade the fstab line is updated accordingly and if the share is actually remounted with the correct options (In reply to Giulio Fidente from comment #12) > (In reply to jritenou from comment #11) > > Hello, > > > > To follow up, I do not have access to the environment I tested with anymore, > > however, I think I have found the issue in comparing notes with others. > > > > In 13, I used this line in my storage config file: > > > > GlanceNfsOptions: 'intr,context=system_u:object_r:glance_var_lib_t:s0' > > > > With 16, it appears the selinux context has changed. > > > > GlanceNfsOptions: rw,sync,context=system_u:object_r:container_file_t:s0 > > note that this is intended; the change comes to fix bz #1742991; we're now > assessing on a new environment if on upgrade the fstab line is updated > accordingly and if the share is actually remounted with the correct options it looks like the ansible tasks we have in host_prep_tasks [1] which are supposed to refresh fstab *and* remount the share on the host (as per ansible module [2]) are actually only updating the fstab content, not remounting the share because ansible tries to do "-o remount" when possible [3] which isn't sufficient to change the NFS mount options 1. https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/glance/glance-api-container-puppet.yaml#L703 2. https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/modules/system/mount.py#L723 3. https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/modules/system/mount.py#L417 *** This bug has been marked as a duplicate of bug 1853433 *** |
Created attachment 1697488 [details] Glance API log from my test environment. Description of problem: NFS backed glance stores provisioned with OSP 13 do not allow read/write after fast forward upgrade to 16.1 due to SELinux permissions issues. Have to set enforcement to permissive on controllers to allow read/write. The controllers themselves mount the glance store fine - it is the glance containers themsleves that are failing to access the store. Version-Release number of selected component (if applicable): 16.1 How reproducible: Always - have obvserved this in two environents at least. Steps to Reproduce: 1. Provision OSP 13 env using director, back glance with NFS and set the necessary selinux context on the store 2. Fast forward upgrade to 16.1 3. Try to upload new image to glance or create instance from existing image Actual results: Glance cannot read/write to the NFS share Expected results: Glance should continue working as expected. Additional info: