Bug 1568120 - Cinder volume incorrectly deployed as both service and container
Summary: Cinder volume incorrectly deployed as both service and container
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
Target Milestone: beta
: 13.0 (Queens)
Assignee: Alan Bishop
QA Contact: Tzach Shefi
Depends On:
Blocks: 1419948 1458798
TreeView+ depends on / blocked
Reported: 2018-04-16 19:10 UTC by Rajini Karthik
Modified: 2018-06-27 13:51 UTC (History)
25 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.2-9.el7ost
Doc Type: Bug Fix
Doc Text:
The Heat templates for Cinder backend services were triggering Puppet to deploy the cinder-volume service on the overcloud host, regardless of whether the service is meant to be deployed in a container. This caused the cinder-volume service to be deployed twice: in a container as well as on the host. Because of this, the OpenStack volume operations (such as creating and attaching a volume) would occasionally fail when the operation was handled by the rogue cinder-volume service running on the host. As a result, the Cinder backend heat templates have been updated to not deploy a second instance of the cinder-volume service.
Clone Of:
Last Closed: 2018-06-27 13:51:11 UTC
Target Upstream Version:
abishop: needinfo-

Attachments (Terms of Use)
containerized cinder (592 bytes, text/plain)
2018-04-16 19:11 UTC, Rajini Karthik
no flags Details
deploy (1016 bytes, text/plain)
2018-04-16 19:12 UTC, Rajini Karthik
no flags Details
env (879 bytes, text/plain)
2018-04-16 19:13 UTC, Rajini Karthik
no flags Details

System ID Private Priority Status Summary Last Updated
Launchpad 1768063 0 None None None 2018-04-30 15:38:57 UTC
OpenStack gerrit 565244 0 'None' MERGED Remove step_config from CinderVolume backend services 2020-08-30 15:54:27 UTC
Red Hat Product Errata RHEA-2018:2086 0 None None None 2018-06-27 13:51:45 UTC

Description Rajini Karthik 2018-04-16 19:10:53 UTC
Description of problem:

With OSP12, cinder volume is tech preview. We had deployed a custom cinder volume container with vnx backend,
Cinder was running as a container AND as a service in the controller node. 
he attachment request was intercepted and processed by the openstack-cinder-volume service on the controller host. 
I shutdown the service and voila everything worked.

The attachment request was intercepted and processed by the openstack-cinder-volume service on the controller host. 

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 1 Rajini Karthik 2018-04-16 19:11:57 UTC
Created attachment 1422699 [details]
containerized cinder

Comment 2 Rajini Karthik 2018-04-16 19:12:40 UTC
Created attachment 1422700 [details]

Comment 3 Rajini Karthik 2018-04-16 19:13:05 UTC
Created attachment 1422701 [details]

Comment 4 Florian Fuchs 2018-04-25 15:58:57 UTC
Rajini, can you explain how this is related to openstack-tripleo-validations?


Comment 5 Rajini Karthik 2018-04-25 16:01:09 UTC
May be not. It is a deployment issue with the director or tipleo may be. I'm not able to change the component, can alan or you change it to the right one

Comment 6 Alan Bishop 2018-04-30 15:38:58 UTC
I confirmed this is an issue that affects all containerized Cinder deployments that use a Cinder backend template in puppet/services, and recommend this be treated as an OSP-13 blocker.

I already have a fix to submit upstream.

Comment 8 Alan Bishop 2018-05-01 19:13:43 UTC
Doc text coming soon.

Comment 14 Avi Avraham 2018-05-02 13:37:23 UTC
This is the systemctl output : 
[root@controller-0 ~]# systemctl status openstack-cinder-volume.service 
● openstack-cinder-volume.service - OpenStack Cinder Volume Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-volume.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@controller-0 ~]# 
[root@controller-0 ~]# 
[root@controller-0 ~]# systemctl status openstack-cinder-scheduler.service 
● openstack-cinder-scheduler.service - OpenStack Cinder Scheduler Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-scheduler.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@controller-0 ~]# 
[root@controller-0 ~]# 
[root@controller-0 ~]# systemctl status openstack-cinder-api.service
● openstack-cinder-api.service - OpenStack Cinder API Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-api.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@controller-0 ~]# 
[root@controller-0 ~]# 
[root@controller-0 ~]# systemctl status openstack-cinder-backup.service 
● openstack-cinder-backup.service - OpenStack Cinder Backup Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-cinder-backup.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

But while running ps -ef I still see the following processes :
[root@controller-0 ~]# docker ps |grep cinder
d85d44d8e98a                  "/bin/bash /usr/lo..."   21 minutes ago      Up 21 minutes                                 openstack-cinder-backup-docker-0
38afad810209                  "/bin/bash /usr/lo..."   22 minutes ago      Up 22 minutes                                 openstack-cinder-volume-docker-0
7618316fd8a7                   "kolla_start"            24 minutes ago      Up 24 minutes                                 cinder_api_cron
cc31ede996d0             "kolla_start"            24 minutes ago      Up 24 minutes (healthy)                       cinder_scheduler
d3bb6b6c55b4                   "kolla_start"            25 minutes ago      Up 25 minutes                                 cinder_api

is it OK ?

Comment 15 Avi Avraham 2018-05-02 13:48:18 UTC
This setup was configured with VNX backend enabled

Comment 16 Alan Bishop 2018-05-02 14:06:04 UTC
Yes, this looks good!

But as we just discussed, it would be great if you could repeat the test on a previous puddle, just to confirm your test methods confirm the problem is visible in a version that doesn't contain the fix.

Comment 17 Avi Avraham 2018-05-03 15:39:02 UTC
RPM package version 
after installation using VNX backend template 
No duplicated Cinder services was enabled and run on the controller
The non relevant parameters in the configuration of cinder.conf have not been added to.

Comment 18 arkady kanevsky 2018-05-03 15:41:12 UTC
Do we need analog BZ for manila and neutron? These are the other 2 components in OSP12 that were left as service.

Comment 19 Rajini Karthik 2018-05-03 16:10:13 UTC
Alan says - he checked the other storage back ends (namely manila) and they are OK. I do not know about non-storage services.

This does not affect Ceph deployments.

Comment 20 Alan Bishop 2018-05-03 16:19:30 UTC
The issue with the cinder backend templates was they "activated" (via puppet-tripleo) the cinder-volume service, and the fix was to remove the lines that did that.

My audit of the manila templates don't indicate a problem. The manila backend templates only contain hiera configuration data, and do not activate any services.

I am less familiar with the neutron templates. It might be appropriate to file a BZ to spur further investigation.

Comment 21 Alan Bishop 2018-05-03 16:25:38 UTC
Looking further, I see there are a number of "dockerized" variations of some neutron templates in docker/services (e.g. docker/services/neutron-plugin-ml2-cisco-vts.yaml). This suggests the neutron team is already  aware of the situation.

Comment 22 arkady kanevsky 2018-05-03 16:27:00 UTC
OK. Sean M. can you check on neutron one?

Comment 26 errata-xmlrpc 2018-06-27 13:51:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.