Bug 1476448 - rhosp-director: OSP12 deployed with cinder and glance on NFS - glance image isn't created in the share.
Summary: rhosp-director: OSP12 deployed with cinder and glance on NFS - glance image i...
Keywords:
Status: CLOSED DUPLICATE of bug 1486995
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 12.0 (Pike)
Assignee: Pranali Deore
QA Contact: Mike Abrams
URL:
Whiteboard:
Depends On: 1486995
Blocks: 1481787 1623968
TreeView+ depends on / blocked
 
Reported: 2017-07-29 06:09 UTC by Alexander Chuzhoy
Modified: 2018-08-30 14:56 UTC (History)
16 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.3-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-15 07:40:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Glance and messages log plus storage yaml (2.78 MB, application/x-gzip)
2017-10-16 09:27 UTC, Tzach Shefi
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 490839 0 None None None 2017-08-04 11:54:40 UTC
OpenStack gerrit 510846 0 None None None 2017-10-16 10:37:53 UTC

Description Alexander Chuzhoy 2017-07-29 06:09:03 UTC
rhosp-director: OSP12 deployed with cinder and glance on NFS - glance image isn't created in the share.

Environment:
instack-undercloud-7.1.1-0.20170714211622.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170721174554.el7ost.noarch

openstack-glance-15.0.0-0.20170718113127.20ea7ab.el7ost.noarch
puppet-glance-11.2.0-0.20170718235959.302415b.el7ost.noarch
python-glanceclient-2.7.0-0.20170622095732.d67b33b.el7ost.noarch
python-glance-store-0.20.1-0.20170719150719.fafc8fd.el7ost.noarch
python-glance-15.0.0-0.20170718113127.20ea7ab.el7ost.noarch



Steps to reproduce:
1. Deploy overcloud with cinder and glance on NFS:

Deployment command:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/templates/nodes_data.yaml \
-e /home/stack/nfs.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/rhos12.yaml \
--log-file overcloud_deployment_0.log


parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsServers: 192.168.24.1:/export/cinder

  GlanceBackend: 'file'
  GlanceNfsEnabled: true
  GlanceNfsShare: '192.168.24.1:/export/glance'


(undercloud) [stack@undercloud-0 ~]$ showmount -e
Export list for undercloud-0.redhat.local:
/export/glance 192.168.24.0/24
/export/cinder 192.168.24.0/24



2.
Try to create glance image and check the dir /export/glance - it's empty.

Note: cinder was created as expected under /export/cinder.



(overcloud) [stack@undercloud-0 ~]$ glance image-list
+--------------------------------------+--------+
| ID                                   | Name   |
+--------------------------------------+--------+
| 63358c06-1cfe-4810-88aa-d078f7e237bc | cirros |
+--------------------------------------+--------+



(overcloud) [stack@undercloud-0 ~]$ glance image-show 63358c06-1cfe-4810-88aa-d078f7e237bc
+------------------+--------------------------------------------------------------------+
| Property         | Value                                                              |
+------------------+--------------------------------------------------------------------+
| checksum         | ee1eca47dc88f4879d8a229cc70a07c6                                   |
| container_format | bare                                                               |
| created_at       | 2017-07-29T05:46:40Z                                               |
| direct_url       | file:///var/lib/glance/images/63358c06-1cfe-4810-88aa-d078f7e237bc |
| disk_format      | qcow2                                                              |
| id               | 63358c06-1cfe-4810-88aa-d078f7e237bc                               |
| min_disk         | 0                                                                  |
| min_ram          | 0                                                                  |
| name             | cirros                                                             |
| owner            | c57730d5f663498eb60e47d747c7d886                                   |
| protected        | False                                                              |
| size             | 13287936                                                           |
| status           | active                                                             |
| tags             | []                                                                 |
| updated_at       | 2017-07-29T05:46:40Z                                               |
| virtual_size     | None                                                               |
| visibility       | shared                                                             |
+------------------+--------------------------------------------------------------------+

Comment 3 Pranali Deore 2017-08-02 12:47:44 UTC
Hi Alexander, 

I'm not having the setup and facing some issues in doing the same, so could you please share your environment details?

Comment 4 Alexander Chuzhoy 2017-08-02 13:09:51 UTC
Hi Pranali,
That setup was reinstalled.

What details you need?

Comment 5 Pranali Deore 2017-08-02 13:51:03 UTC
I just wanted to check 

1. "mount" command details on controller node. 

2. docker inspect <container-id of glance-api> output.

Comment 6 Alexander Chuzhoy 2017-08-02 19:58:56 UTC
(undercloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks               |
+--------------------------------------+------------------------+--------+------------+-------------+------------------------+
| 2a189b62-1323-4a37-93a6-dfc3f9f91b4a | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.168.24.17 |
| db707d7f-2b92-4878-8749-3727bdff72be | overcloud-compute-1    | ACTIVE | -          | Running     | ctlplane=192.168.24.10 |
| fa847161-a509-4d56-8ba6-56786d25f210 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.12 |
| ab896c37-5d26-4f8d-affd-0fbfeb910d21 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.168.24.8  |
| 9914c75c-e565-4c25-9237-30996a44551a | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.168.24.15 |
+--------------------------------------+------------------------+--------+------------+-------------+------------------------+


Mount:


(undercloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.12 "sudo mount"
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3975648k,nr_inodes=993912,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/vda2 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12732)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
tmpfs on /run/user/978 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700,uid=978,gid=974)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
192.168.24.1:/export/cinder on /var/lib/cinder/mnt/884306f0cb952b10bf6c43194652b6da type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.24.12,local_lock=none,addr=192.168.24.1)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700,uid=1000,gid=1000)
##############################################################################


(undercloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.8 "sudo mount"
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3975648k,nr_inodes=993912,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/vda2 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11224)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
tmpfs on /run/user/978 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700,uid=978,gid=974)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700,uid=1000,gid=1000)
###########################################################################


(undercloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.15 "sudo mount"
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3975648k,nr_inodes=993912,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/vda2 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10985)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
tmpfs on /run/user/978 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700,uid=978,gid=974)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=801020k,mode=700,uid=1000,gid=1000)
(undercloud) [stack@undercloud-0 ~]$ 



Docker inspect:
[root@overcloud-controller-2 ~]# sudo docker ps|awk '/glance_api/ {print $1}'
b4fbcef9c209
[root@overcloud-controller-2 ~]# docker inspect b4fbcef9c209
[
    {
        "Id": "b4fbcef9c209b80f992a59c089d2e52d41eac244e3559c21e5bb382be6266a77",
        "Created": "2017-08-02T18:53:44.348025841Z",
        "Path": "kolla_start",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 87890,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-08-02T18:53:44.818205347Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:eea39202c91c91634f6a8f4df5979ef731170ea932655dec9ff5030939fd8cbc",
        "ResolvConfPath": "/var/lib/docker/containers/b4fbcef9c209b80f992a59c089d2e52d41eac244e3559c21e5bb382be6266a77/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/b4fbcef9c209b80f992a59c089d2e52d41eac244e3559c21e5bb382be6266a77/hostname",
        "HostsPath": "/etc/hosts",
        "LogPath": "",
        "Name": "/glance_api",
        "RestartCount": 0,
        "Driver": "overlay2",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/lib/config-data/puppet-generated/glance_api/:/var/lib/kolla/config_files/src:ro",
                "/var/log/containers/glance:/var/log/glance",
                "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
                "/dev/log:/dev/log",
                "/etc/hosts:/etc/hosts:ro",
                "/etc/localtime:/etc/localtime:ro",
                "/etc/puppet:/etc/puppet:ro",
                "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
                "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
                "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
                "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
                "/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "docker-runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/75095e57beb7fefb9681ca5855af08611c601eee0192e8fff7dfcfd6151a7374-init/diff:/var/lib/docker/overlay2/e9766b1db118a455a15e8741793b4a00e6b60b0cf5053e1148d7c48e94c121e3/diff:/var/lib/docker/overlay2/b4d3236a028038ec553a3b5c6f1378239d51d331a8c54e686f8e6fa485fe1932/diff:/var/lib/docker/overlay2/fe242598ea799e8857dcc1c338f0b52fdce7bf5bf2e1e2d90a198e23459f175a/diff:/var/lib/docker/overlay2/5c861640999c86747a3f91a464ba3add891eddb3a3638e429a8d2a3143fa2478/diff:/var/lib/docker/overlay2/3e3dbde88e77fc75fe46f492ef9c25ee8a2a5fae0eb65029812f9d65ca885a17/diff:/var/lib/docker/overlay2/be3d8ca9542ad83fb1e755b5b1a9356bf76f90791dcca0e679c8a35f0a874812/diff",
                "MergedDir": "/var/lib/docker/overlay2/75095e57beb7fefb9681ca5855af08611c601eee0192e8fff7dfcfd6151a7374/merged",
                "UpperDir": "/var/lib/docker/overlay2/75095e57beb7fefb9681ca5855af08611c601eee0192e8fff7dfcfd6151a7374/diff",
                "WorkDir": "/var/lib/docker/overlay2/75095e57beb7fefb9681ca5855af08611c601eee0192e8fff7dfcfd6151a7374/work"
            }
        },
        "Mounts": [
            {
                "Source": "/etc/hosts",
                "Destination": "/etc/hosts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/localtime",
                "Destination": "/etc/localtime",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/puppet",
                "Destination": "/etc/puppet",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/certs/ca-bundle.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.crt",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/kolla/config_files/glance_api.json",
                "Destination": "/var/lib/kolla/config_files/config.json",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/config-data/puppet-generated/glance_api",
                "Destination": "/var/lib/kolla/config_files/src",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/cert.pem",
                "Destination": "/etc/pki/tls/cert.pem",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/dev/log",
                "Destination": "/dev/log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/ssh/ssh_known_hosts",
                "Destination": "/etc/ssh/ssh_known_hosts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/log/containers/glance",
                "Destination": "/var/log/glance",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/ca-trust/extracted",
                "Destination": "/etc/pki/ca-trust/extracted",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "overcloud-controller-2.redhat.local",
            "Domainname": "",
            "User": "glance",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=oci",
                "KOLLA_BASE_DISTRO=rhel",
                "KOLLA_INSTALL_TYPE=binary",
                "KOLLA_INSTALL_METATYPE=rhos",
                "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ "
            ],
            "Cmd": [
                "kolla_start"
            ],
            "Image": "192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-07-26.10",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "Kolla-SHA": "5.0.0.0b2-195-gec16e33d",
                "architecture": "x86_64",
                "authoritative-source-url": "registry.access.redhat.com",
                "build-date": "2017-07-26T15:55:42.896775",
                "com.redhat.build-host": "ip-10-29-120-106.ec2.internal",
                "com.redhat.component": "openstack-glance-api-docker",
                "config_data": "{\"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"start_order\": 2, \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated/glance_api/:/var/lib/kolla/config_files/src:ro\", \"/var/log/containers/glance:/var/log/glance\"], \"image\": \"192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-07-26.10\", \"net\": \"host\", \"restart\": \"always\", \"privileged\": false}",
                "config_id": "tripleo_step4",
                "container_name": "glance_api",
                "description": "The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.",
                "distribution-scope": "public",
                "io.k8s.description": "The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.",
                "io.k8s.display-name": "Red Hat Enterprise Linux 7",
                "io.openshift.tags": "base rhel7",
                "kolla_version": "master",
                "managed_by": "paunch",
                "name": "rhosp12/rhel-binary-glance-api",
                "release": "20170726.1",
                "summary": "Provides the latest release of Red Hat Enterprise Linux 7 in a fully featured and supported base image.",
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/rhel-binary-glance-api/images/12.0-20170726.1",
                "vcs-ref": "1ddc9136b3205c317d9d811c3f9ad71513597213",
                "vcs-type": "git",
                "vendor": "Red Hat, Inc.",
                "version": "12.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "b6aa498acdd930656edba815207f2302ad37ae1bc52dd1b93fa36ac4c448cbfc",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "f4f38595f73a66737ae04c3059684ac07e96d6e435a397cb49830dab244087fd",
                    "EndpointID": "20c31e8e881265e5e471e3f9e70a9803939ffd23082c5cf83b80ce90dd982b54",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]


#############################################################################


[root@overcloud-controller-1 ~]# docker ps|awk '/glance_api/ {print $1}'
0bf60affc14d
[root@overcloud-controller-1 ~]# docker inspect 0bf60affc14d
[
    {
        "Id": "0bf60affc14d256a9a71055d09bcc552e184964f75d381020224b431a0898d6c",
        "Created": "2017-08-02T18:53:55.607793418Z",
        "Path": "kolla_start",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 89140,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-08-02T18:53:56.145912981Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:eea39202c91c91634f6a8f4df5979ef731170ea932655dec9ff5030939fd8cbc",
        "ResolvConfPath": "/var/lib/docker/containers/0bf60affc14d256a9a71055d09bcc552e184964f75d381020224b431a0898d6c/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/0bf60affc14d256a9a71055d09bcc552e184964f75d381020224b431a0898d6c/hostname",
        "HostsPath": "/etc/hosts",
        "LogPath": "",
        "Name": "/glance_api",
        "RestartCount": 0,
        "Driver": "overlay2",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
                "/dev/log:/dev/log",
                "/var/lib/config-data/puppet-generated/glance_api/:/var/lib/kolla/config_files/src:ro",
                "/etc/hosts:/etc/hosts:ro",
                "/etc/puppet:/etc/puppet:ro",
                "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
                "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
                "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
                "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
                "/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json",
                "/var/log/containers/glance:/var/log/glance",
                "/etc/localtime:/etc/localtime:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "docker-runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/f1c4c15cbea3c84f9e9730b74f7c31dbf1b0c63754d0a56287a79f13e0f6b2ef-init/diff:/var/lib/docker/overlay2/243b66cffc4183e9158f3ffbb205bd213998ec1e5f2dbf91cebc6d8c1c4ccf1a/diff:/var/lib/docker/overlay2/93a7a00392780a3afd76641a1b8348f8d7490cfe89e1d63fab5aa0f1dffbbc54/diff:/var/lib/docker/overlay2/5f3f82db05f73b464b43c9bbbc2982a44657aaa2831c342e878468a1fda8c572/diff:/var/lib/docker/overlay2/b0457286daff9c11283ba751f42d9003671675b8f12f1f4a25c3b910fbc68fe0/diff:/var/lib/docker/overlay2/7cd09a06d700a53d69dc192195c9832f8f69686e09f5e9e8886567e52034788a/diff:/var/lib/docker/overlay2/734cf47d01c968c89dba6272b5d962c5e54eef9b3e982152e4fe9ccefd3f35b8/diff",
                "MergedDir": "/var/lib/docker/overlay2/f1c4c15cbea3c84f9e9730b74f7c31dbf1b0c63754d0a56287a79f13e0f6b2ef/merged",
                "UpperDir": "/var/lib/docker/overlay2/f1c4c15cbea3c84f9e9730b74f7c31dbf1b0c63754d0a56287a79f13e0f6b2ef/diff",
                "WorkDir": "/var/lib/docker/overlay2/f1c4c15cbea3c84f9e9730b74f7c31dbf1b0c63754d0a56287a79f13e0f6b2ef/work"
            }
        },
        "Mounts": [
            {
                "Source": "/etc/ssh/ssh_known_hosts",
                "Destination": "/etc/ssh/ssh_known_hosts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/kolla/config_files/glance_api.json",
                "Destination": "/var/lib/kolla/config_files/config.json",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/log/containers/glance",
                "Destination": "/var/log/glance",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/cert.pem",
                "Destination": "/etc/pki/tls/cert.pem",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/config-data/puppet-generated/glance_api",
                "Destination": "/var/lib/kolla/config_files/src",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/hosts",
                "Destination": "/etc/hosts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/puppet",
                "Destination": "/etc/puppet",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/certs/ca-bundle.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.crt",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/localtime",
                "Destination": "/etc/localtime",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/ca-trust/extracted",
                "Destination": "/etc/pki/ca-trust/extracted",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/dev/log",
                "Destination": "/dev/log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "overcloud-controller-1.redhat.local",
            "Domainname": "",
            "User": "glance",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=oci",
                "KOLLA_BASE_DISTRO=rhel",
                "KOLLA_INSTALL_TYPE=binary",
                "KOLLA_INSTALL_METATYPE=rhos",
                "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ "
            ],
            "Cmd": [
                "kolla_start"
            ],
            "Image": "192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-07-26.10",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "Kolla-SHA": "5.0.0.0b2-195-gec16e33d",
                "architecture": "x86_64",
                "authoritative-source-url": "registry.access.redhat.com",
                "build-date": "2017-07-26T15:55:42.896775",
                "com.redhat.build-host": "ip-10-29-120-106.ec2.internal",
                "com.redhat.component": "openstack-glance-api-docker",
                "config_data": "{\"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"start_order\": 2, \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated/glance_api/:/var/lib/kolla/config_files/src:ro\", \"/var/log/containers/glance:/var/log/glance\"], \"image\": \"192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-07-26.10\", \"net\": \"host\", \"restart\": \"always\", \"privileged\": false}",
                "config_id": "tripleo_step4",
                "container_name": "glance_api",
                "description": "The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.",
                "distribution-scope": "public",
                "io.k8s.description": "The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.",
                "io.k8s.display-name": "Red Hat Enterprise Linux 7",
                "io.openshift.tags": "base rhel7",
                "kolla_version": "master",
                "managed_by": "paunch",
                "name": "rhosp12/rhel-binary-glance-api",
                "release": "20170726.1",
                "summary": "Provides the latest release of Red Hat Enterprise Linux 7 in a fully featured and supported base image.",
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/rhel-binary-glance-api/images/12.0-20170726.1",
                "vcs-ref": "1ddc9136b3205c317d9d811c3f9ad71513597213",
                "vcs-type": "git",
                "vendor": "Red Hat, Inc.",
                "version": "12.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "acc3c2ceb23be4f458e38def836e37b2532433b945f542776e37a20dee74996f",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "38a7a6b9036d1fce74d60aa53eaefe584dcfd8406030bced796f07bcd3d33f3b",
                    "EndpointID": "8bbd815d2f4262d96d5f4664e109cd3da90598de025c1b04e5a0b1782065615f",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]


############################################################################


[root@overcloud-controller-0 ~]# docker ps|awk '/glance_api/ {print $1}'
0fbb534024b6
[root@overcloud-controller-0 ~]# docker inspect 0fbb534024b6
[
    {
        "Id": "0fbb534024b6fe6213a63500676f1d2288c0f0a0a6779f717f788b4fd2ca643b",
        "Created": "2017-08-02T18:54:21.023614899Z",
        "Path": "kolla_start",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 99144,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-08-02T18:54:21.933026104Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:eea39202c91c91634f6a8f4df5979ef731170ea932655dec9ff5030939fd8cbc",
        "ResolvConfPath": "/var/lib/docker/containers/0fbb534024b6fe6213a63500676f1d2288c0f0a0a6779f717f788b4fd2ca643b/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/0fbb534024b6fe6213a63500676f1d2288c0f0a0a6779f717f788b4fd2ca643b/hostname",
        "HostsPath": "/etc/hosts",
        "LogPath": "",
        "Name": "/glance_api",
        "RestartCount": 0,
        "Driver": "overlay2",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
                "/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json",
                "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
                "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
                "/dev/log:/dev/log",
                "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
                "/etc/hosts:/etc/hosts:ro",
                "/etc/localtime:/etc/localtime:ro",
                "/etc/puppet:/etc/puppet:ro",
                "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
                "/var/lib/config-data/puppet-generated/glance_api/:/var/lib/kolla/config_files/src:ro",
                "/var/log/containers/glance:/var/log/glance"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "docker-runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/b53b8b059bdfb18d8a0c46e02ba7fe303c084fe7057a86bca868cd74b9a2d3e8-init/diff:/var/lib/docker/overlay2/e4731a88d001f9ffd2b3767d29e2d0fda8a1126ccf727585dd9c169f6d1561a9/diff:/var/lib/docker/overlay2/35cf1fa7d2428f2c0651171abd3608ea21865a847ae9dab15448ca7aca5c206a/diff:/var/lib/docker/overlay2/9d8643551aaa19b4fb2674414f29e9363053356fbbbcb7de6a627d2d11d74afa/diff:/var/lib/docker/overlay2/052141507d188e7e4958f9148a53992d59f5f0cab443a3e9599fa6ac65fd35ae/diff:/var/lib/docker/overlay2/982b1aae00217b35ee5c8358bc09d4c89d1a3731ff7054c292d3427b71cecafe/diff:/var/lib/docker/overlay2/0ad9f414b46b568face58ab1e8380728749241aedceecfaed27534cf1c940644/diff",
                "MergedDir": "/var/lib/docker/overlay2/b53b8b059bdfb18d8a0c46e02ba7fe303c084fe7057a86bca868cd74b9a2d3e8/merged",
                "UpperDir": "/var/lib/docker/overlay2/b53b8b059bdfb18d8a0c46e02ba7fe303c084fe7057a86bca868cd74b9a2d3e8/diff",
                "WorkDir": "/var/lib/docker/overlay2/b53b8b059bdfb18d8a0c46e02ba7fe303c084fe7057a86bca868cd74b9a2d3e8/work"
            }
        },
        "Mounts": [
            {
                "Source": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/kolla/config_files/glance_api.json",
                "Destination": "/var/lib/kolla/config_files/config.json",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/cert.pem",
                "Destination": "/etc/pki/tls/cert.pem",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/dev/log",
                "Destination": "/dev/log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/localtime",
                "Destination": "/etc/localtime",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/puppet",
                "Destination": "/etc/puppet",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/config-data/puppet-generated/glance_api",
                "Destination": "/var/lib/kolla/config_files/src",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/tls/certs/ca-bundle.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.crt",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/ssh/ssh_known_hosts",
                "Destination": "/etc/ssh/ssh_known_hosts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/hosts",
                "Destination": "/etc/hosts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/etc/pki/ca-trust/extracted",
                "Destination": "/etc/pki/ca-trust/extracted",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/log/containers/glance",
                "Destination": "/var/log/glance",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "overcloud-controller-0.redhat.local",
            "Domainname": "",
            "User": "glance",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=oci",
                "KOLLA_BASE_DISTRO=rhel",
                "KOLLA_INSTALL_TYPE=binary",
                "KOLLA_INSTALL_METATYPE=rhos",
                "PS1=$(tput bold)($(printenv KOLLA_SERVICE_NAME))$(tput sgr0)[$(id -un)@$(hostname -s) $(pwd)]$ "
            ],
            "Cmd": [
                "kolla_start"
            ],
            "Image": "192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-07-26.10",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "Kolla-SHA": "5.0.0.0b2-195-gec16e33d",
                "architecture": "x86_64",
                "authoritative-source-url": "registry.access.redhat.com",
                "build-date": "2017-07-26T15:55:42.896775",
                "com.redhat.build-host": "ip-10-29-120-106.ec2.internal",
                "com.redhat.component": "openstack-glance-api-docker",
                "config_data": "{\"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"start_order\": 2, \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated/glance_api/:/var/lib/kolla/config_files/src:ro\", \"/var/log/containers/glance:/var/log/glance\"], \"image\": \"192.168.24.1:8787/rhosp12/openstack-glance-api-docker:2017-07-26.10\", \"net\": \"host\", \"restart\": \"always\", \"privileged\": false}",
                "config_id": "tripleo_step4",
                "container_name": "glance_api",
                "description": "The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.",
                "distribution-scope": "public",
                "io.k8s.description": "The Red Hat Enterprise Linux Base image is designed to be a fully supported foundation for your containerized applications.  This base image provides your operations and application teams with the packages, language runtimes and tools necessary to run, maintain, and troubleshoot all of your applications. This image is maintained by Red Hat and updated regularly. It is designed and engineered to be the base layer for all of your containerized applications, middleware and utilites. When used as the source for all of your containers, only one copy will ever be downloaded and cached in your production environment. Use this image just like you would a regular Red Hat Enterprise Linux distribution. Tools like yum, gzip, and bash are provided by default. For further information on how this image was built look at the /root/anacanda-ks.cfg file.",
                "io.k8s.display-name": "Red Hat Enterprise Linux 7",
                "io.openshift.tags": "base rhel7",
                "kolla_version": "master",
                "managed_by": "paunch",
                "name": "rhosp12/rhel-binary-glance-api",
                "release": "20170726.1",
                "summary": "Provides the latest release of Red Hat Enterprise Linux 7 in a fully featured and supported base image.",
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/rhel-binary-glance-api/images/12.0-20170726.1",
                "vcs-ref": "1ddc9136b3205c317d9d811c3f9ad71513597213",
                "vcs-type": "git",
                "vendor": "Red Hat, Inc.",
                "version": "12.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "f3150606841811397d5d27e22ae1b4fad245daaff1ed3719d26af43759ca72a3",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "080b1d83cf1a7fe698bce420eb80961766b4b90425912e3ae6e0b94699a49293",
                    "EndpointID": "5ae43e5c18caf06f9665253b98f472c4e0a6774ef5749bf63d8b8446e8d1049c",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]

Comment 7 Pranali Deore 2017-08-04 11:53:55 UTC
This was happening because NFS volume was not mounted on docker container.

Comment 8 Paul Grist 2017-08-10 14:56:23 UTC
Patch is upstream for review

Comment 9 Pranali Deore 2017-08-18 08:45:30 UTC
Patch is merged in upstream.

Comment 12 Alexander Chuzhoy 2017-08-31 02:36:20 UTC
This bug depends on https://bugzilla.redhat.com/show_bug.cgi?id=1486995

Comment 13 Tzach Shefi 2017-10-16 09:27:51 UTC
Created attachment 1339112 [details]
Glance and messages log plus storage yaml

Failing to verify this. 
Notice #12 if this depends on a bug which is still in post state(ops12). 
How can this bug be ON_QA at (osp11)? 

Cinder mount is created fine, but Glance's mount isn't 
Happened twice, I've tested with nfs.yaml as Alexander did and also with my own twist ( using mnt option -> nosharecache  bz1491597) both fail. 

Glance api log

2017-10-15 15:33:26.132 1 ERROR glance_store._drivers.filesystem [-] Unable to create datadir: /var/lib/glance/images/: OSError: [Errno 13] Permission denied: '/var/lib/glance/images/'
2017-10-15 15:33:26.133 1 WARNING glance_store.driver [-] Failed to configure store correctly: Store filesystem could not be configured correctly. Reason: Unable to create datadir: /var/lib/glance/images/ Disabling add method.: BadStoreConfiguration: Store filesystem could not be configured correctly. Reason: Unable to create datadir: /var/lib/glance/images/

Same error as mentioned on Alexander's bz 1486995

Comment 15 Giulio Fidente 2017-10-16 10:37:53 UTC
I think you guys are missing https://review.openstack.org/#/c/510846 to see this passing. Moving it back to ON_DEV

Comment 18 Pranali Deore 2017-11-03 04:54:50 UTC
Patch[1] is merged in upstream.

[1]: https://review.openstack.org/#/c/510846/1

Comment 19 Mike Abrams 2017-11-15 07:40:35 UTC
duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1486995

*** This bug has been marked as a duplicate of bug 1486995 ***


Note You need to log in before you can comment on or make changes to this bug.