[root@overcloud-controller-0 ~]# ls -la /var/lib/glance/
total 4
drwxr-xr-x.  2 glance nobody    6 Aug 25 14:48 .
drwxr-xr-x. 88 root   root   4096 Aug 31 01:33 ..
[root@overcloud-controller-0 ~]# getent passwd glance
glance:x:161:161:OpenStack Glance Daemons:/var/lib/glance:/sbin/nologin
[root@overcloud-controller-0 ~]# docker exec -u root -it glance_api bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
()[root@overcloud-controller-0 /]# ls -la /var/lib/glance/
total 0
drwxr-xr-x. 2  161 nobody  6 Aug 25 14:48 .
drwxr-xr-x. 1 root root   19 Aug 31 01:50 ..
()[root@overcloud-controller-0 /]# getent passwd glance
glance:x:42415:42415::/var/lib/glance:/usr/sbin/nologin

Leaving for PTO in -30 minutes but i gave this a glance ;) so i'll post my findings at least...

docker/services/glance-api.yaml will need to be amended to create and chown the directory similarly as we do for /var/log/glance. Other than that, mounting of the NFS endpoint for Glance is not handled by the service itself (unlike in Cinder). We've previously done the mounting for Glance via Puppet, so probably an additional change may be needed make sure that the mount happens. This may be conceptually similar situation to br-ex creation, and a similar solution might apply [2].

[1] https://github.com/openstack/puppet-tripleo/blob/11cec1c6f1db1c29543c3f187780b99722a79bc7/manifests/glance/nfs_mount.pp
[2] https://github.com/openstack/tripleo-heat-templates/blob/70718ff4caf05a71cc13598e0cc0107341edce0d/docker/services/neutron-ovs-agent.yaml#L81-L111

Jiri, Thanks for investigation :) I missed your comments otherwise I could have saved some time to find the permission issue.
  
Apart from the permissoin issues for mounted NFS volume there are few more issues here.

The image still would not be created in the share as the NFS endpoint is not getting mounted successfully on the container. .
I have investigated little more on this and noticed that the mounting of NFS endpoint[1] doesn't work inside the container even though it makes mountpoint entry in /etc/fstab.

"----------------------------------------------------------------------------------------------------------------------------
()[glance@overcloud-controller-0 /]$ cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Tue Aug  1 17:23:07 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=1af758b4-34e7-40f6-a12d-9ee83d3d521e /                       ext4    defaults        1 1
192.168.24.1:/export/glance /var/lib/glance/images nfs4 _netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0 0 0
----------------------------------------------------------------------------------------------------------------------------"

I wonder, why the OC deployment doesn't fail when it fails to mount NFS endpoint on container or it dosen't even try to mount? (I couldn't find anything related to it in logs :()
I've found only this much logs related to nfs, mounting endpoint is missing:

'''
Sep  7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure) created
Sep  7 15:23:56 localhost puppet-user[10]: (/Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure) created
Sep  7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File[/var/lib/glance/images]/ensure: created
Sep  7 15:23:56 localhost journal: Notice: /Stage[main]/Tripleo::Glance::Nfs_mount/File_line[NFS for glance in fstab]/ensure: created

'''

I'm new in tripleo and don't have hands-on knowledge and experience on containers but after doing some research on this,
I suspect this mounting NFS endpoint on container might not be happening due to below reasons:
1. nfs-utils is not installed in the container
2. container is not privileged and it runs with non-root user(glance)
3. additionally I think, auto mount for NFS volume (entry in fstab) doesn't work on container restart


IMO, any of the following could be used for fixing this issue (Please correct me if I'm wrong)

1. mount NFS endpoint on the host ( adding mount in host-prep-tasks section), and fix the permission issue as pointed by Jiri
2. Create glance_api container with nfs-utils and run it in privileged mode with root user(in this case, mounted NFS volume will not be needed) and we would need to modify kolla start to mount entries in fstab
3. fall back to non-containerized similar to cinder[2].
4. or could be possible by using the docker volume NFS plugin but I haven't tried this yet.



[1]: https://github.com/openstack/puppet-tripleo/blob/07500da43e6a991511099a37106015037da159b8/manifests/glance/nfs_mount.pp#L66-L70
[2]: https://bugs.launchpad.net/tripleo/+bug/1698136

Erno, need your suggestion on this..

I've proposed a patch with first solution in upstream i.e., mounting NFS endpoint on host.

It looks like patch https://review.openstack.org/#/c/502403/ is merged, do you still need more info or is this ready to go into "POST" if the fix is now merged upstream?

No, seems there is nothing wrong with this solution :P, I'm glad, it's got merged so quickly ..:)

Same issue on osp11
https://bugzilla.redhat.com/show_bug.cgi?id=1476448#c13

Maybe we should duplicate this bz for osp11? 

Glance api reports same OSError 

Versions
python-glanceclient-2.8.0-0.20170811141216.52eb529.el7ost.noarch
puppet-glance-11.3.1-0.20170830102216.b3bb1db.el7ost.noarch
python-glance-store-0.22.0-0.20170821160800.49c915f.el7ost.noarch
python-glance-15.0.1-0.20170920231317.9091d26.el7ost.noarch
openstack-glance-15.0.1-0.20170920231317.9091d26.el7ost.noarch

instack-undercloud-7.4.1-0.20170925172804.el7ost.noarch
openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch
openstack-tripleo-heat-templates-7.0.1-0.20170925173114.el7ost.1.noarch

Storage yaml
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderBackupBackend: swift
  CinderEnableNfsBackend: true
  NovaEnableRbdBackend: false
  GlanceBackend: file
  GnocchiBackend: swift


  #### CINDER NFS SETTINGS ####

  ## NFS mount options
  CinderNfsMountOptions: 'nosharecache'
  # NFS mount point, e.g. '192.168.122.1:/export/cinder'
  CinderNfsServers: '10.35.160.111:/export/ins_cinder'


  #### GLANCE NFS SETTINGS ####

  ## Make sure to set `GlanceBackend: file` when enabling NFS
  ##
  ## Whether to make Glance 'file' backend a NFS mount
  GlanceNfsEnabled: true
  ## NFS share for image storage, e.g. '192.168.122.1:/export/glance'
  ## (If using IPv6, use both double- and single-quotes,
  ## e.g. "'[fdd0::1]:/export/glance'")
  GlanceNfsShare: '10.35.160.111:/export/ins_glance'
  ## Mount options for the NFS image storage mount point
  GlanceNfsOptions: 'nosharecache'


Glance api error

2017-10-15 15:33:26.132 1 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'
2017-10-15 15:33:26.133 1 WARNING glance_store.driver [-] Failed to configure store correctly: Store filesystem could not be configured correctly. Reason: Unable to create datadir: /var/lib/glance/images/ Disabling add method.: BadStoreConfiguration: Store filesystem could not be configured correctly. Reason: Unable to create datadir: /var/lib/glance/images/


[root@controller-0 ~]# docker inspect glance_api|jq ".[].Mounts"|grep "/var/lib/glance"
    "Destination": "/var/lib/glance",
    "Source": "/var/lib/glance"


[root@controller-0 ~]# ls -la /var/lib/glance/
total 4
drwxr-xr-x.  2 glance nobody    6 ספט 26 07:50 .
drwxr-xr-x. 87 root   root   4096 אוק 15 15:14 ..
[root@controller-0 ~]# getent passwd glance
glance:x:161:161:OpenStack Glance Daemons:/var/lib/glance:/sbin/nologin
[root@controller-0 ~]# docker exec -u root -it glance_api bash
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
()[root@controller-0 /]# ls -la /var/lib/glance/
total 0
drwxr-xr-x. 2  161 nobody  6 Sep 26 07:50 .
drwxr-xr-x. 1 root root   19 Oct 15 15:33 ..
()[root@controller-0 /]# getent passwd glance
glance:x:42415:42415::/var/lib/glance:/usr/sbin/nologin

If puppet is mounting the nfs export it will only be mounted in the short lived puppet config container.

See https://review.openstack.org/518179 & https://review.openstack.org/518180. I needed this for nfs mounts to propagate from the nova_compute container to the host & nova_libvirt container.

FailedQA

Environment:
openstack-tripleo-heat-templates-7.0.3-0.20171024200823.el7ost.noarch


Unable to create glance image and still see: 
2017-11-07 15:49:43.389 1 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'

stable/pike patch is merged can you look and move back to POST if that is is?

Can you confirm the stable pike patch covers this?  If so can you move this back to "POST"

https://review.openstack.org/#/c/510846/

Sounds like the recent test may not have had the fix?

Yes, It's got missed in the latest import :(
So, now it's there in rhos-12.0-patches branch but still not there in latest puddle.

openstack-tripleo-heat-templates-7.0.3-3.el7ost - is built and contains the patch

Verified:

Environment:  openstack-tripleo-heat-templates-7.0.3-3.el7ost.noarch

The reported issue didn't reproduce.
Was able to create glance image on NFS share.


(undercloud) [stack@undercloud-0 ~]$ . keystonerc_master 
(overcloud) [stack@undercloud-0 ~]$ openstack image list
openstack image s+--------------------------------------+--------+--------+
| ID                                   | Name   | Status |
+--------------------------------------+--------+--------+
| 95bf4398-3762-4603-bd60-46b9070ddfbc | cirros | active |
+--------------------------------------+--------+--------+
(overcloud) [stack@undercloud-0 ~]$ openstack image show cirros
+------------------+---------------------------------------------------------------------------------+
| Field            | Value                                                                           |
+------------------+---------------------------------------------------------------------------------+
| checksum         | ee1eca47dc88f4879d8a229cc70a07c6                                                |
| container_format | bare                                                                            |
| created_at       | 2017-11-15T02:11:07Z                                                            |
| disk_format      | qcow2                                                                           |
| file             | /v2/images/95bf4398-3762-4603-bd60-46b9070ddfbc/file                            |
| id               | 95bf4398-3762-4603-bd60-46b9070ddfbc                                            |
| min_disk         | 0                                                                               |
| min_ram          | 0                                                                               |
| name             | cirros                                                                          |
| owner            | b52f3563dc3f4a0f8f6328ee32239f32                                                |
| properties       | direct_url='file:///var/lib/glance/images/95bf4398-3762-4603-bd60-46b9070ddfbc' |
| protected        | False                                                                           |
| schema           | /v2/schemas/image                                                               |
| size             | 13287936                                                                        |
| status           | active                                                                          |
| tags             |                                                                                 |
| updated_at       | 2017-11-15T02:11:08Z                                                            |
| virtual_size     | None                                                                            |
| visibility       | shared                                                                          |
+------------------+---------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.13
Last login: Wed Nov 15 04:04:33 2017 from 192.168.24.1

[heat-admin@overcloud-controller-0 ~]$ sudo df /var/lib/glance/images/95bf4398-3762-4603-bd60-46b9070ddfbc
Filesystem                  1K-blocks     Used Available Use% Mounted on
192.168.24.1:/export/glance  57659392 27744256  29915136  49% /var/lib/glance

*** Bug 1476448 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462