Bug 1658367 - Glance store for cinder fails on PrivSep
Summary: Glance store for cinder fails on PrivSep
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-glance
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Abhishek Kekane
QA Contact: Mike Abrams
Kim Nylander
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-11 21:07 UTC by Rajini Karthik
Modified: 2019-04-29 12:35 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-11 19:50:24 UTC
Target Upstream Version:
tshefi: automate_bug-


Attachments (Terms of Use)
docker inspect (22.35 KB, text/plain)
2019-01-09 22:52 UTC, Rajini Karthik
no flags Details

Description Rajini Karthik 2018-12-11 21:07:15 UTC
Description of problem:
Glance store for cinder fails on PrivSep

Glance-api: logs

2018-12-07 16:50:39.787 33 INFO oslo.privsep.daemon [req-6d67bf6e-0343-467f-bba7-63e813591071 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Running privsep helper: ['sudo', 'glance-rootwrap', '/etc/glance/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/glance/glance-api-dist.conf', '--config-file', '/etc/glance/glance-api.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpO4RHAQ/privsep.sock']
2018-12-07 16:50:40.337 33 WARNING oslo.privsep.daemon [-] privsep log: [Errno 1] Operation not permitted
2018-12-07 16:50:40.377 1 WARNING glance.common.wsgi [-] Unrecognised child 371
2018-12-07 16:50:40.378 1 ERROR glance.common.wsgi [-] Not respawning child 371, cannot recover from termination
2018-12-07 16:50:40.388 33 INFO oslo.privsep.daemon [req-6d67bf6e-0343-467f-bba7-63e813591071 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Spawned new privsep daemon via rootwrap
2018-12-07 16:50:40.330 371 INFO oslo.privsep.daemon [-] privsep daemon starting
2018-12-07 16:50:40.333 371 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2018-12-07 16:50:40.336 371 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted

FailedToDropPrivileges: Privsep daemon failed to start
1:21 PM ERROR glance_store._drivers.cinder [req-6d67bf6e-0343-467f-bba7-63e813591071 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Failed to write to volume 35f6f6e8-74cb-40e2-a7a4-ae2c75a0d8e6.: FailedToDropPrivileges: Privsep daemon failed to start

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 9 Rajini Karthik 2019-01-08 18:01:00 UTC
I have added the privsep to the rootwrap, but I'm suspecting that the container is not built with SYS_ADMIN capability, privsep daemon can't start

vi /var/lib/config-data/glance_api/etc/glance/rootwrap.d/glance_cinder_store.filters
# os-brick library commands
# os_brick.privileged.run_as_root oslo.privsep context
# This line ties the superuser privs with the config files, context name,
# and (implicitly) the actual python code invoked.
privsep-rootwrap: RegExpFilter, privsep-helper, root, privsep-helper, --config-file, /etc/(?!\.\.).*, --privsep_context, os_brick.privileged.default, --privsep_sock_path, /tmp/.*


But the log from glance-api show that,the Privsep daemon failed to start.


2019-01-08 17:31:38.586 26 INFO oslo.privsep.daemon [req-89134cd0-cd51-47d7-8a11-68ec6404d8d0 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Running privsep helper: ['sudo', 'glance-rootwrap', '/etc/glance/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/glance/glance-api-dist.conf', '--config-file', '/etc/glance/glance-api.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpzM6bLo/privsep.sock']
2019-01-08 17:31:39.201 26 WARNING oslo.privsep.daemon [-] privsep log: [Errno 1] Operation not permitted
2019-01-08 17:31:39.227 1 WARNING glance.common.wsgi [-] Unrecognised child 74
2019-01-08 17:31:39.230 1 ERROR glance.common.wsgi [-] Not respawning child 74, cannot recover from termination
2019-01-08 17:31:39.236 26 INFO oslo.privsep.daemon [req-89134cd0-cd51-47d7-8a11-68ec6404d8d0 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Spawned new privsep daemon via rootwrap
2019-01-08 17:31:39.187 74 INFO oslo.privsep.daemon [-] privsep daemon starting
2019-01-08 17:31:39.190 74 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2019-01-08 17:31:39.192 74 ERROR oslo.privsep.daemon [-] [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 508, in helper_main
    Daemon(channel, context).run()
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 360, in run
    self._drop_privs()
  File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 396, in _drop_privs
    capabilities.drop_all_caps_except(self.caps, self.caps, [])
  File "/usr/lib/python2.7/site-packages/oslo_privsep/capabilities.py", line 156, in drop_all_caps_except
    raise OSError(errno, os.strerror(errno))
OSError: [Errno 1] Operation not permitted
2019-01-08 17:31:39.238 26 ERROR oslo.privsep.daemon [req-89134cd0-cd51-47d7-8a11-68ec6404d8d0 300e912f27664756ba49b4db572afab6 a37da43e6ec949088e4398ba46c499f1 - default default] Error while sending initial PING to privsep: [Errno 32] Broken pipe: error: [Errno 32] Broken pipe

Comment 10 Rajini Karthik 2019-01-08 18:50:10 UTC
See https://review.openstack.org/#/c/506033/

Comment 12 arkady kanevsky 2019-01-08 21:49:01 UTC
Rajini,
are you saying that you expect that we need
  capabilities:
              add: ["SYS_ADMIN"]

For Cinder container?
See https://review.openstack.org/#/c/506033/ is for cinder_backup.

Comment 13 Rajini Karthik 2019-01-08 22:30:19 UTC
For the glance container. Maybe. I'm not sure if that will fix the problem. May be there is a easy solution to update the rootwrap filter

Comment 16 Rajini Karthik 2019-01-09 19:15:49 UTC
There is a support case opened - https://access.redhat.com/support/cases/#/case/02275165

Can you share your environment details to make this work

1. The OSP version? I'm using OSP13/queens
2. The glance.conf
3. What cinder backend is being used? This is important. Is it rbd? We are using Dell SC.
4. Share the glance.cinder.rootwrap file
5. How are the containers started? Are we using heat-admin user or root in the controllers? This might play a role because we are seeing permission issues with rootwrap? In my environment the heat-admin, but I restarted the containers as root using "docker restart glance_api"

Comment 17 Alan Bishop 2019-01-09 20:52:36 UTC
Rajini, please run "sudo docker inspect glance_api" on the controller and attach the output to this bz.

Comment 18 Rajini Karthik 2019-01-09 22:52:40 UTC
Created attachment 1519601 [details]
docker inspect

Comment 19 Rajini Karthik 2019-01-09 22:53:04 UTC
Attached the docker inspect output

Comment 20 Alan Bishop 2019-01-10 16:05:33 UTC
Rajini,

The data is very insightful, and what I'm seeing is NOT consistent with what you should get when the deployment includes "GlanceBackend: cinder". Specifically, the container does not have the settings necessary for accessing cinder via iSCSI, and one of the settings is likely the cause of your privsep issue.

Take a look at [1] and note where "cinder_backend_enabled" appears elsewhere in that file. You should be able to see how it affects the way the glance-api container runs.

[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/glance-api.yaml#L96

Are you including "GlanceBackend: cinder" in your deployment?

Comment 21 Rajini Karthik 2019-01-11 14:50:26 UTC
I was looking at your comment ""GlanceBackend: cinder" for deployment. I didn't do anything special during deployment. Only changed the glance-api.conf to use the cinder as default_store and restarted the containers
Is that not sufficient?

Comment 22 Alan Bishop 2019-01-11 15:03:00 UTC
Sorry, no, that's not sufficient.

Setting "GlanceBackend: cinder" adds two critical settings that are used when the glance-api container is started:

1) Two docker volume mounts required to use the iSCSI service [1]
2) Run the container in privileged mode [2], which is also required to access iSCSI services

The second item is what I believe will eliminate your privsep failure, but you need both.

[1] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/glance-api.yaml#L197
[2] https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/glance-api.yaml#L210

Comment 23 Rajini Karthik 2019-01-11 19:50:24 UTC
As per alan's guidance, updated the deployment using by setting the GlanceBackend: cinder tripleo parameter. Now I can create images and glance is using the cinder store without any errors.

Verified
1. glance-api.conf has cinder set as default store
2. glance-api.conf has cinder added to the stores


(overcloud) [stack@director etc]$ glance image-show b1adb119-74de-4180-97ea-d62703a18d76
+------------------+-----------------------------------------------+
| Property         | Value                                         |
+------------------+-----------------------------------------------+
| checksum         | ee1eca47dc88f4879d8a229cc70a07c6              |
| container_format | bare                                          |
| created_at       | 2019-01-11T19:22:39Z                          |
| direct_url       | cinder://05da61f4-3a64-47be-8dc9-dbef6d45a1f8 |
| disk_format      | qcow2                                         |
| id               | b1adb119-74de-4180-97ea-d62703a18d76          |
| min_disk         | 0                                             |
| min_ram          | 0                                             |
| name             | cirros_via_cinder_1                           |
| owner            | a37da43e6ec949088e4398ba46c499f1              |
| protected        | False                                         |
| size             | 13287936                                      |
| status           | active                                        |
| tags             | []                                            |
| updated_at       | 2019-01-11T19:23:00Z                          |
| virtual_size     | None                                          |
| visibility       | public                                        |
+------------------+-----------------------------------------------+



(overcloud) [stack@director etc]$ cinder show image-b1adb119-74de-4180-97ea-d62703a18d76
+--------------------------------+--------------------------------------------------------+
| Property                       | Value                                                  |
+--------------------------------+--------------------------------------------------------+
| attached_servers               | []                                                     |
| attachment_ids                 | []                                                     |
| availability_zone              | nova                                                   |
| bootable                       | false                                                  |
| consistencygroup_id            | None                                                   |
| created_at                     | 2019-01-11T19:22:41.000000                             |
| description                    | None                                                   |
| encrypted                      | False                                                  |
| id                             | 05da61f4-3a64-47be-8dc9-dbef6d45a1f8                   |
| metadata                       | glance_image_id : b1adb119-74de-4180-97ea-d62703a18d76 |
|                                | image_owner : a37da43e6ec949088e4398ba46c499f1         |
|                                | image_size : 13287936                                  |
|                                | readonly : True                                        |
| migration_status               | None                                                   |
| multiattach                    | False                                                  |
| name                           | image-b1adb119-74de-4180-97ea-d62703a18d76             |
| os-vol-host-attr:host          | hostgroup@dellsc#dellsc                                |
| os-vol-mig-status-attr:migstat | None                                                   |
| os-vol-mig-status-attr:name_id | None                                                   |
| os-vol-tenant-attr:tenant_id   | a37da43e6ec949088e4398ba46c499f1                       |
| readonly                       | True                                                   |
| replication_status             | None                                                   |
| size                           | 1                                                      |
| snapshot_id                    | None                                                   |
| source_volid                   | None                                                   |
| status                         | available                                              |
| updated_at                     | 2019-01-11T19:22:55.000000                             |
| user_id                        | 300e912f27664756ba49b4db572afab6                       |
| volume_type                    | dellsc_backend                                         |
+--------------------------------+--------------------------------------------------------+

Not a bug, this can be resolved

Comment 24 Paul Grist 2019-01-11 20:05:47 UTC
Thanks for the update and including the output, great to see :)

Comment 25 Tzach Shefi 2019-04-29 12:35:54 UTC
Closed not a bug, nothing to test/automate per close loop.


Note You need to log in before you can comment on or make changes to this bug.