Bug 1486995 - rhosp-director: OC deployed with NFS: unable to create glance image: ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'
Summary: rhosp-director: OC deployed with NFS: unable to create glance image: ERROR ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 12.0 (Pike)
Assignee: Pranali Deore
QA Contact: Alexander Chuzhoy
Don Domingo
URL:
Whiteboard:
: 1476448 (view as bug list)
Depends On:
Blocks: 1476448 1481787 1623968
TreeView+ depends on / blocked
 
Reported: 2017-08-31 02:33 UTC by Alexander Chuzhoy
Modified: 2018-08-30 14:56 UTC (History)
20 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.3-3.el7ost
Doc Type: Known Issue
Doc Text:
When using an NFS back end for the Image service (glance), attempting to create an image will fail with a permission error. This is because the user ID on the host and container differ, and also because puppet cannot mount the NFS endpoint successfully on the container.
Clone Of:
Environment:
Last Closed: 2017-12-13 21:59:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 502403 None MERGED Providing required priviledges to the mounted NFS volume 2020-08-25 07:55:18 UTC
OpenStack gerrit 510846 None MERGED Providing required priviledges to the mounted NFS volume 2020-08-25 07:55:18 UTC
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Alexander Chuzhoy 2017-08-31 02:33:59 UTC
rhosp-director: OC deployed with NFS:  unable to create glance image:  ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'

Environment:

python-glance-15.0.0-0.20170821194716.1610cda.el7ost.noarch
openstack-glance-15.0.0-0.20170821194716.1610cda.el7ost.noarch
python-glance-store-0.22.0-0.20170821160800.49c915f.el7ost.noarch
python-glanceclient-2.8.0-0.20170811141216.52eb529.el7ost.noarch
puppet-glance-11.3.0-0.20170805095742.851ee9d.el7ost.noarch

instack-undercloud-7.2.1-0.20170821194210.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170712001959.0333c73.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost.noarch


Steps to reproduce:
Deploy overlcoud with:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/templates/nodes_data.yaml \
-e /home/stack/nfs.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/rhos12.yaml


(undercloud) [stack@undercloud-0 ~]$ cat nfs.yaml 
parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: 192.168.24.1:/export/cinder

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsShare: '192.168.24.1:/export/glance'





Try to create an image in glance:

(overcloud) [stack@undercloud-0 ~]$ openstack image list
openstack image show +--------------------------------------+--------+--------+
| ID                                   | Name   | Status |
+--------------------------------------+--------+--------+
| 4df0cdcc-db0c-4751-8b84-176f108df044 | cirros | queued |
+--------------------------------------+--------+--------+
(overcloud) [stack@undercloud-0 ~]$ openstack image show cirros
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| checksum         | None                                                 |
| container_format | bare                                                 |
| created_at       | 2017-08-31T01:58:01Z                                 |
| disk_format      | qcow2                                                |
| file             | /v2/images/4df0cdcc-db0c-4751-8b84-176f108df044/file |
| id               | 4df0cdcc-db0c-4751-8b84-176f108df044                 |
| min_disk         | 0                                                    |
| min_ram          | 0                                                    |
| name             | cirros                                               |
| owner            | 1d7b50f22f314ed8a0d25ee2cfb0052b                     |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | None                                                 |
| status           | queued                                               |
| tags             |                                                      |
| updated_at       | 2017-08-31T01:58:03Z                                 |
| virtual_size     | None                                                 |
| visibility       | shared                                               |
+------------------+------------------------------------------------------+


Looking for errors in /var/log/containers/glance/api.log:


2017-08-31 01:50:08.613 1 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'



[root@overcloud-controller-0 ~]# docker inspect glance_api|jq ".[].Mounts"|grep "/var/lib/glance"
    "Destination": "/var/lib/glance",
    "Source": "/var/lib/glance"



No images dir in /var/lib/glance/

Comment 2 Alexander Chuzhoy 2017-08-31 15:30:20 UTC
[root@overcloud-controller-0 ~]# ls -la /var/lib/glance/
total 4
drwxr-xr-x.  2 glance nobody    6 Aug 25 14:48 .
drwxr-xr-x. 88 root   root   4096 Aug 31 01:33 ..
[root@overcloud-controller-0 ~]# getent passwd glance
glance:x:161:161:OpenStack Glance Daemons:/var/lib/glance:/sbin/nologin
[root@overcloud-controller-0 ~]# docker exec -u root -it glance_api bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
()[root@overcloud-controller-0 /]# ls -la /var/lib/glance/
total 0
drwxr-xr-x. 2  161 nobody  6 Aug 25 14:48 .
drwxr-xr-x. 1 root root   19 Aug 31 01:50 ..
()[root@overcloud-controller-0 /]# getent passwd glance
glance:x:42415:42415::/var/lib/glance:/usr/sbin/nologin

Comment 3 Jiri Stransky 2017-08-31 16:30:42 UTC
Leaving for PTO in -30 minutes but i gave this a glance ;) so i'll post my findings at least...

docker/services/glance-api.yaml will need to be amended to create and chown the directory similarly as we do for /var/log/glance. Other than that, mounting of the NFS endpoint for Glance is not handled by the service itself (unlike in Cinder). We've previously done the mounting for Glance via Puppet, so probably an additional change may be needed make sure that the mount happens. This may be conceptually similar situation to br-ex creation, and a similar solution might apply [2].

[1] https://github.com/openstack/puppet-tripleo/blob/11cec1c6f1db1c29543c3f187780b99722a79bc7/manifests/glance/nfs_mount.pp
[2] https://github.com/openstack/tripleo-heat-templates/blob/70718ff4caf05a71cc13598e0cc0107341edce0d/docker/services/neutron-ovs-agent.yaml#L81-L111

Comment 4 Pranali Deore 2017-09-08 07:03:59 UTC
Jiri, Thanks for investigation :) I missed your comments otherwise I could have saved some time to find the permission issue.
  
Apart from the permissoin issues for mounted NFS volume there are few more issues here.

The image still would not be created in the share as the NFS endpoint is not getting mounted successfully on the container. .
I have investigated little more on this and noticed that the mounting of NFS endpoint[1] doesn't work inside the container even though it makes mountpoint entry in /etc/fstab.

"----------------------------------------------------------------------------------------------------------------------------
()[glance@overcloud-controller-0 /]$ cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Aug  1 17:23:07 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=1af758b4-34e7-40f6-a12d-9ee83d3d521e /                       ext4    defaults        1 1
192.168.24.1:/export/glance /var/lib/glance/images nfs4 _netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0
----------------------------------------------------------------------------------------------------------------------------"

I wonder, why the OC deployment doesn't fail when it fails to mount NFS endpoint on container or it dosen't even try to mount? (I couldn't find anything related to it in logs :()
I've found only this much logs related to nfs, mounting endpoint is missing:

'''
Sep  7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure) created
Sep  7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure) created
Sep  7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure: created
Sep  7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure: created

'''

I'm new in tripleo and don't have hands-on knowledge and experience on containers but after doing some research on this,
I suspect this mounting NFS endpoint on container might not be happening due to below reasons:
1. nfs-utils is not installed in the container
2. container is not privileged and it runs with non-root user(glance)
3. additionally I think, auto mount for NFS volume (entry in fstab) doesn't work on container restart


IMO, any of the following could be used for fixing this issue (Please correct me if I'm wrong)

1. mount NFS endpoint on the host ( adding mount in host-prep-tasks section), and fix the permission issue as pointed by Jiri
2. Create glance_api container with nfs-utils and run it in privileged mode with root user(in this case, mounted NFS volume will not be needed) and we would need to modify kolla start to mount entries in fstab
3. fall back to non-containerized similar to cinder[2].
4. or could be possible by using the docker volume NFS plugin but I haven't tried this yet.



[1]: https://github.com/openstack/puppet-tripleo/blob/07500da43e6a991511099a37106015037da159b8/manifests/glance/nfs_mount.pp#L66-L70
[2]: https://bugs.launchpad.net/tripleo/+bug/1698136

Erno, need your suggestion on this..

Comment 5 Pranali Deore 2017-09-11 10:21:12 UTC
I've proposed a patch with first solution in upstream i.e., mounting NFS endpoint on host.

Comment 6 Paul Grist 2017-09-13 00:33:28 UTC
It looks like patch https://review.openstack.org/#/c/502403/ is merged, do you still need more info or is this ready to go into "POST" if the fix is now merged upstream?

Comment 7 Pranali Deore 2017-09-13 14:14:16 UTC
No, seems there is nothing wrong with this solution :P, I'm glad, it's got merged so quickly ..:)

Comment 8 Tzach Shefi 2017-10-16 09:46:54 UTC
Same issue on osp11
https://bugzilla.redhat.com/show_bug.cgi?id=1476448#c13

Maybe we should duplicate this bz for osp11? 

Glance api reports same OSError 

Versions
python-glanceclient-2.8.0-0.20170811141216.52eb529.el7ost.noarch
puppet-glance-11.3.1-0.20170830102216.b3bb1db.el7ost.noarch
python-glance-store-0.22.0-0.20170821160800.49c915f.el7ost.noarch
python-glance-15.0.1-0.20170920231317.9091d26.el7ost.noarch
openstack-glance-15.0.1-0.20170920231317.9091d26.el7ost.noarch

instack-undercloud-7.4.1-0.20170925172804.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch
openstack-tripleo-heat-templates-7.0.1-0.20170925173114.el7ost.1.noarch

Storage yaml
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderBackupBackend: swift
  CinderEnableNfsBackend: true
  NovaEnableRbdBackend: false
  GlanceBackend: file
  GnocchiBackend: swift


  #### CINDER NFS SETTINGS ####

  ## NFS mount options
  CinderNfsMountOptions: 'nosharecache'
  # NFS mount point, e.g. '192.168.122.1:/export/cinder'
  CinderNfsServers: '10.35.160.111:/export/ins_cinder'


  #### GLANCE NFS SETTINGS ####

  ## Make sure to set `GlanceBackend: file` when enabling NFS
  ##
  ## Whether to make Glance 'file' backend a NFS mount
  GlanceNfsEnabled: true
  ## NFS share for image storage, e.g. '192.168.122.1:/export/glance'
  ## (If using IPv6, use both double- and single-quotes,
  ## e.g. "'[fdd0::1]:/export/glance'")
  GlanceNfsShare: '10.35.160.111:/export/ins_glance'
  ## Mount options for the NFS image storage mount point
  GlanceNfsOptions: 'nosharecache'


Glance api error

2017-10-15 15:33:26.132 1 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'
2017-10-15 15:33:26.133 1 WARNING glance_store.driver [-] Failed to configure store correctly: Store filesystem could not be configured correctly. Reason: Unable to create datadir: /var/lib/glance/images/ Disabling add method.: BadStoreConfiguration: Store filesystem could not be configured correctly. Reason: Unable to create datadir: /var/lib/glance/images/


[root@controller-0 ~]# docker inspect glance_api|jq ".[].Mounts"|grep "/var/lib/glance"
    "Destination": "/var/lib/glance",
    "Source": "/var/lib/glance"


[root@controller-0 ~]# ls -la /var/lib/glance/
total 4
drwxr-xr-x.  2 glance nobody    6 ספט 26 07:50 .
drwxr-xr-x. 87 root   root   4096 אוק 15 15:14 ..
[root@controller-0 ~]# getent passwd glance
glance:x:161:161:OpenStack Glance Daemons:/var/lib/glance:/sbin/nologin
[root@controller-0 ~]# docker exec -u root -it glance_api bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
()[root@controller-0 /]# ls -la /var/lib/glance/
total 0
drwxr-xr-x. 2  161 nobody  6 Sep 26 07:50 .
drwxr-xr-x. 1 root root   19 Oct 15 15:33 ..
()[root@controller-0 /]# getent passwd glance
glance:x:42415:42415::/var/lib/glance:/usr/sbin/nologin

Comment 13 Ollie Walsh 2017-11-07 16:01:42 UTC
If puppet is mounting the nfs export it will only be mounted in the short lived puppet config container.

See https://review.openstack.org/518179 & https://review.openstack.org/518180. I needed this for nfs mounts to propagate from the nova_compute container to the host & nova_libvirt container.

Comment 14 Alexander Chuzhoy 2017-11-07 16:06:10 UTC
FailedQA

Environment:
openstack-tripleo-heat-templates-7.0.3-0.20171024200823.el7ost.noarch


Unable to create glance image and still see: 
2017-11-07 15:49:43.389 1 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'

Comment 15 Jon Schlueter 2017-11-08 13:59:23 UTC
stable/pike patch is merged can you look and move back to POST if that is is?

Comment 16 Paul Grist 2017-11-13 22:12:09 UTC
Can you confirm the stable pike patch covers this?  If so can you move this back to "POST"

https://review.openstack.org/#/c/510846/

Sounds like the recent test may not have had the fix?

Comment 17 Pranali Deore 2017-11-14 12:02:53 UTC
Yes, It's got missed in the latest import :(
So, now it's there in rhos-12.0-patches branch but still not there in latest puddle.

Comment 18 Jon Schlueter 2017-11-14 12:10:28 UTC
openstack-tripleo-heat-templates-7.0.3-3.el7ost - is built and contains the patch

Comment 21 Alexander Chuzhoy 2017-11-15 04:06:02 UTC
Verified:

Environment:  openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch

The reported issue didn't reproduce.
Was able to create glance image on NFS share.


(undercloud) [stack@undercloud-0 ~]$ . keystonerc_master 
(overcloud) [stack@undercloud-0 ~]$ openstack image list
openstack image s+--------------------------------------+--------+--------+
| ID                                   | Name   | Status |
+--------------------------------------+--------+--------+
| 95bf4398-3762-4603-bd60-46b9070ddfbc | cirros | active |
+--------------------------------------+--------+--------+
(overcloud) [stack@undercloud-0 ~]$ openstack image show cirros
+------------------+---------------------------------------------------------------------------------+
| Field            | Value                                                                           |
+------------------+---------------------------------------------------------------------------------+
| checksum         | ee1eca47dc88f4879d8a229cc70a07c6                                                |
| container_format | bare                                                                            |
| created_at       | 2017-11-15T02:11:07Z                                                            |
| disk_format      | qcow2                                                                           |
| file             | /v2/images/95bf4398-3762-4603-bd60-46b9070ddfbc/file                            |
| id               | 95bf4398-3762-4603-bd60-46b9070ddfbc                                            |
| min_disk         | 0                                                                               |
| min_ram          | 0                                                                               |
| name             | cirros                                                                          |
| owner            | b52f3563dc3f4a0f8f6328ee32239f32                                                |
| properties       | direct_url='file:///var/lib/glance/images/95bf4398-3762-4603-bd60-46b9070ddfbc' |
| protected        | False                                                                           |
| schema           | /v2/schemas/image                                                               |
| size             | 13287936                                                                        |
| status           | active                                                                          |
| tags             |                                                                                 |
| updated_at       | 2017-11-15T02:11:08Z                                                            |
| virtual_size     | None                                                                            |
| visibility       | shared                                                                          |
+------------------+---------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@192.168.24.13
Last login: Wed Nov 15 04:04:33 2017 from 192.168.24.1

[heat-admin@overcloud-controller-0 ~]$ sudo df /var/lib/glance/images/95bf4398-3762-4603-bd60-46b9070ddfbc
Filesystem                  1K-blocks     Used Available Use% Mounted on
192.168.24.1:/export/glance  57659392 27744256  29915136  49% /var/lib/glance

Comment 22 Mike Abrams 2017-11-15 07:40:35 UTC
*** Bug 1476448 has been marked as a duplicate of this bug. ***

Comment 25 errata-xmlrpc 2017-12-13 21:59:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.