I have added the privsep to the rootwrap, but I'm suspecting that the container is not built with SYS_ADMIN capability, privsep daemon can't start

vi /var/lib/config-data/glance_api/etc/glance/rootwrap.d/glance_cinder_store.filters
# os-brick library commands
# os_brick.privileged.run_as_root oslo.privsep context
# This line ties the superuser privs with the config files, context name,
# and (implicitly) the actual python code invoked.
privsep-rootwrap: RegExpFilter, privsep-helper, root, privsep-helper, --config-file, /etc/(?!\.\.).*, --privsep_context, os_brick.privileged.default, --privsep_sock_path, /tmp/.*


But the log from glance-api show that,the Privsep daemon failed to start.


2019-01-08 17:31:38.586 26 INFO oslo.privsep.daemon [req-89134cd0-cd51-47d7-8a11-68ec6404d8d0 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Running privsep helper: ['sudo', 'glance-rootwrap', '/etc/glance/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/glance/glance-api-dist.conf', '--config-file', '/etc/glance/glance-api.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpzM6bLo/privsep.sock']
2019-01-08 17:31:39.201 26 WARNING oslo.privsep.daemon [-] privsep log: [Errno 1] Operation not permitted
2019-01-08 17:31:39.227 1 WARNING glance.common.wsgi [-] Unrecognised child 74
2019-01-08 17:31:39.230 1 ERROR glance.common.wsgi [-] Not respawning child 74, cannot recover from termination
2019-01-08 17:31:39.236 26 INFO oslo.privsep.daemon [req-89134cd0-cd51-47d7-8a11-68ec6404d8d0 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Spawned new privsep daemon via rootwrap
2019-01-08 17:31:39.187 74 INFO oslo.privsep.daemon [-] privsep daemon starting
2019-01-08 17:31:39.190 74 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2019-01-08 17:31:39.192 74 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 508, in helper_main
    Daemon(channel, context).run()
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 360, in run
    self._drop_privs()
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 396, in _drop_privs
    capabilities.drop_all_caps_except(self.caps, self.caps, [])
  File "/usr/lib/python2.7/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
    raise OSError(errno, os.strerror(errno))
OSError: [Errno 1] Operation not permitted
2019-01-08 17:31:39.238 26 ERROR oslo.privsep.daemon [req-89134cd0-cd51-47d7-8a11-68ec6404d8d0 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Error while sending initial PING to privsep: [Errno 32] Broken pipe: error: [Errno 32] Broken pipe

See https://review.openstack.org/#/c/506033/

Rajini,
are you saying that you expect that we need
  capabilities:
              add: ["SYS_ADMIN"]

For Cinder container?
See https://review.openstack.org/#/c/506033/ is for cinder_backup.

For the glance container. Maybe. I'm not sure if that will fix the problem. May be there is a easy solution to update the rootwrap filter

There is a support case opened - https://access.redhat.com/support/cases/#/case/02275165

Can you share your environment details to make this work

1. The OSP version? I'm using OSP13/queens
2. The glance.conf
3. What cinder backend is being used? This is important. Is it rbd? We are using Dell SC.
4. Share the glance.cinder.rootwrap file
5. How are the containers started? Are we using heat-admin user or root in the controllers? This might play a role because we are seeing permission issues with rootwrap? In my environment the heat-admin, but I restarted the containers as root using "docker restart glance_api"

Rajini, please run "sudo docker inspect glance_api" on the controller and attach the output to this bz.

Created attachment 1519601 [details]
docker inspect

Attached the docker inspect output

Rajini,

The data is very insightful, and what I'm seeing is NOT consistent with what you should get when the deployment includes "GlanceBackend: cinder". Specifically, the container does not have the settings necessary for accessing cinder via iSCSI, and one of the settings is likely the cause of your privsep issue.

Take a look at [1] and note where "cinder_backend_enabled" appears elsewhere in that file. You should be able to see how it affects the way the glance-api container runs.

[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/glance-api.yaml#L96

Are you including "GlanceBackend: cinder" in your deployment?

I was looking at your comment ""GlanceBackend: cinder" for deployment. I didn't do anything special during deployment. Only changed the glance-api.conf to use the cinder as default_store and restarted the containers
Is that not sufficient?

Sorry, no, that's not sufficient.

Setting "GlanceBackend: cinder" adds two critical settings that are used when the glance-api container is started:

1) Two docker volume mounts required to use the iSCSI service [1]
2) Run the container in privileged mode [2], which is also required to access iSCSI services

The second item is what I believe will eliminate your privsep failure, but you need both.

[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/glance-api.yaml#L197
[2] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/glance-api.yaml#L210

As per alan's guidance, updated the deployment using by setting the GlanceBackend: cinder tripleo parameter. Now I can create images and glance is using the cinder store without any errors.

Verified
1. glance-api.conf has cinder set as default store
2. glance-api.conf has cinder added to the stores


(overcloud) [stack@director etc]$ glance image-show b1adb119-74de-4180-97ea-d62703a18d76
+------------------+-----------------------------------------------+
| Property         | Value                                         |
+------------------+-----------------------------------------------+
| checksum         | ee1eca47dc88f4879d8a229cc70a07c6              |
| container_format | bare                                          |
| created_at       | 2019-01-11T19:22:39Z                          |
| direct_url       | cinder://05da61f4-3a64-47be-8dc9-dbef6d45a1f8 |
| disk_format      | qcow2                                         |
| id               | b1adb119-74de-4180-97ea-d62703a18d76          |
| min_disk         | 0                                             |
| min_ram          | 0                                             |
| name             | cirros_via_cinder_1                           |
| owner            | a37da43e6ec949088e4398ba46c499f1              |
| protected        | False                                         |
| size             | 13287936                                      |
| status           | active                                        |
| tags             | []                                            |
| updated_at       | 2019-01-11T19:23:00Z                          |
| virtual_size     | None                                          |
| visibility       | public                                        |
+------------------+-----------------------------------------------+



(overcloud) [stack@director etc]$ cinder show image-b1adb119-74de-4180-97ea-d62703a18d76
+--------------------------------+--------------------------------------------------------+
| Property                       | Value                                                  |
+--------------------------------+--------------------------------------------------------+
| attached_servers               | []                                                     |
| attachment_ids                 | []                                                     |
| availability_zone              | nova                                                   |
| bootable                       | false                                                  |
| consistencygroup_id            | None                                                   |
| created_at                     | 2019-01-11T19:22:41.000000                             |
| description                    | None                                                   |
| encrypted                      | False                                                  |
| id                             | 05da61f4-3a64-47be-8dc9-dbef6d45a1f8                   |
| metadata                       | glance_image_id : b1adb119-74de-4180-97ea-d62703a18d76 |
|                                | image_owner : a37da43e6ec949088e4398ba46c499f1         |
|                                | image_size : 13287936                                  |
|                                | readonly : True                                        |
| migration_status               | None                                                   |
| multiattach                    | False                                                  |
| name                           | image-b1adb119-74de-4180-97ea-d62703a18d76             |
| os-vol-host-attr:host          | hostgroup@dellsc#dellsc                                |
| os-vol-mig-status-attr:migstat | None                                                   |
| os-vol-mig-status-attr:name_id | None                                                   |
| os-vol-tenant-attr:tenant_id   | a37da43e6ec949088e4398ba46c499f1                       |
| readonly                       | True                                                   |
| replication_status             | None                                                   |
| size                           | 1                                                      |
| snapshot_id                    | None                                                   |
| source_volid                   | None                                                   |
| status                         | available                                              |
| updated_at                     | 2019-01-11T19:22:55.000000                             |
| user_id                        | 300e912f27664756ba49b4db572afab6                       |
| volume_type                    | dellsc_backend                                         |
+--------------------------------+--------------------------------------------------------+

Not a bug, this can be resolved

Thanks for the update and including the output, great to see :)

Closed not a bug, nothing to test/automate per close loop.