Bug 1464108 - [Openstack-containers][OSP12]: cinder volume create is failing when cinder-volume and cinder-backup are running within containers.
[Openstack-containers][OSP12]: cinder volume create is failing when cinder-vo...
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers (Show other bugs)
12.0 (Pike)
x86_64 Linux
medium Severity medium
: ga
: 12.0 (Pike)
Assigned To: Jon Bernard
Omri Hochman
Andrew Burden
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-22 08:40 EDT by Omri Hochman
Modified: 2017-07-26 13:43 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Omri Hochman 2017-06-22 08:40:17 EDT
[Openstack-containers][OSP12]: cinder volume create is failing when cinder-volume and cinder-backup are running within containers. 


Steps
-------
- run deployment of osp12 with containers ( 1 controller , 1 compute )
- run ping test using 
(1) mkdir /home/stack/tripleo/ 
(2) cd /home/stack/tripleo/ && git clone https://github.com/openstack-infra/tripleo-ci
(3) source /home/stack/overcloudrc && bash /home/stack/tripleo/tripleo-ci/scripts/tripleo.sh --overcloud-pingtest

environment: 
-------------
openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost.noarch


heat-engine.log:
----------------
7-06-22 10:20:53.759 1 WARNING heat.engine.environment [-] OS::Neutron::LoadBalancer is HIDDEN. Use LBaaS V2 instead.
2017-06-22 10:20:53.759 1 WARNING heat.engine.environment [-] OS::Neutron::PoolMember is HIDDEN. Use LBaaS V2 instead.
...skipping...
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource Traceback (most recent call last):
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 808, in _action_recorder
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     yield
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1814, in delete
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     *action_args)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 351, in wrapper
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     step = next(subtask)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 867, in action_handler_task
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     done = check(handler_data)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/volume_base.py", line 143, in check_delete_complete
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     prg.delete['complete'] = self._delete_volume()
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/volume_base.py", line 129, in _delete_volume
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     self.client_plugin().ignore_not_found(ex)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 342, in __call__
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     six.reraise(exc_type, exc_val, traceback)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/volume_base.py", line 126, in _delete_volume
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     cinder.volumes.delete(self.resource_id)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 314, in delete
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     return self._delete(loc)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 332, in _delete
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     resp, body = self.api.client.delete(url)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 184, in delete
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     return self._cs_request(url, 'DELETE', **kwargs)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 166, in _cs_request
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     return self.request(url, method, **kwargs)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 155, in request
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource     raise exceptions.from_response(resp, body)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource BadRequest: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group or have snapshots. (HTTP 400) (Request-ID: req-58345541-955c-40f5-8d1d-0edad585116c)
2017-06-22 10:46:31.218 26 ERROR heat.engine.resource
2017-06-22 10:46:31.269 26 INFO heat.engine.stack [req-f799c3a7-c90a-438f-b4b5-8050cdf5c601 - admin - - -] Stack DELETE FAILED (tenant-stack): Resource DELETE failed: BadRequest: resources.volume1: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group or have snapshots. (HTTP 400) (Request-ID: req-58345541-955c-40f5-8d1d-0edad585116c): ResourceFailure: BadRequest: resources.volume1: Invalid volume: Volume status must be avail:
Comment 3 Omri Hochman 2017-06-22 08:53:38 EDT
workaround : 
-------------
run those services not on containers by removing:

 OS::TripleO::Services::CinderBackup: ../docker/services/cinder-backup.yaml
 OS::TripleO::Services::CinderVolume: ../docker/services/cinder-volume.yaml


from -> /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml


** before starting the overcloud deployment.
Comment 5 Dan Prince 2017-07-11 13:36:38 EDT
Omri, Martin, and I spoke about this today and we think this might still need to be addressed for OSP12 as we still intend to deliver a working Cinder (and Cinder HA) solution for OSP12 that would allow a partner to integrate with as a tech preview feature.

So even though we are going to default the Cinder services to baremetal for OSP12 we'll want to have docs on how to flip them to containers and at least some basic (non partner) containerized support for them to integrate with. Including the HA bits as well.
Comment 7 Paul Grist 2017-07-18 18:18:05 EDT
We may need help from the folks who worked on the HA patches for cinder here and the first step may be to ensure all the cinder container patches have landed.

Ack on cinder will ship defaulted to BM for OSP-12, so this would not be a release blocker.

However, we need to verify containerized Cinder as a tech preview and this should be covered by a test case for that.

Lowering the sev/prio as we expect downstream changes soon to default cinder and manila to BM and this one can be worked around in the meantime.  Feel free to comment if that doesn't seem like a reasonable approach for now.

Note You need to log in before you can comment on or make changes to this bug.