Bug 1303093 - [RFE] Add ability to disable Swift from overcloud deployment
[RFE] Add ability to disable Swift from overcloud deployment
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
7.0 (Kilo)
All Linux
unspecified Severity medium
: rc
: 10.0 (Newton)
Assigned To: Giulio Fidente
Mike Abrams
: FutureFeature, Triaged
: 1322985 (view as bug list)
Depends On: 1357551
Blocks: 1335593 1356451 1320209
  Show dependency treegraph
 
Reported: 2016-01-29 09:21 EST by Jon Jozwiak
Modified: 2017-04-05 03:40 EDT (History)
41 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-5.0.0-0.20160929150845.4cdc4fc.el7ost
Doc Type: Enhancement
Doc Text:
With this update, it is possible to disable the Object Storage service (swift) in the overcloud by using an additional environment file when deploying the overcloud. The environment file should contain the following: resource_registry: OS::TripleO::Services::SwiftProxy: OS::Heat::None OS::TripleO::Services::SwiftStorage: OS::Heat::None OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None As a result, the Object Storage service will not be running in the overcloud and there will not be an endpoint for the Object Storage service in the overcloud Identity service.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-14 10:21:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
scohen: needinfo+


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
OpenStack gerrit 353307 None None None 2016-08-29 08:33 EDT

  None (edit)
Description Jon Jozwiak 2016-01-29 09:21:15 EST
Description of problem:

Today the default deployment of OSP 7 deploys Swift Proxy on the controller nodes as well as loopback devices for storage on the controllers.  For customers who do not want object storage, there is no way to disable Swift.  At best you can disable deploying the loopback devices on the controllers.  However, this will just result in a Swift proxy with no backend

Version-Release number of selected component (if applicable):
RHEL OSP 7 (all versions) 

How reproducible:
This can be reproduced by doing a default deployment of the overcloud from Director.  


Expected results:
An 'enable swift' flag should be given in the controller that allows you to disable deployment of Swift proxy.  

Additional info:
Comment 3 Steven Hardy 2016-03-24 12:30:49 EDT
It should already be possible to disable the services directly via puppet ExtraConfig overrides, something like this (untested atm):

$ cat disable_swift_env.yaml
parameter_defaults:
  controllerExtraConfig:
    swift::proxy::enabled: false
    swift::service::enabled: false

openstack overcloud deploy --templates -e disable_swift_env.yaml

Basically this wires in a boolean directly to the puppet hieradata, which will ovetrride the defaults here:

https://github.com/openstack/puppet-swift/blob/master/manifests/proxy.pp#L85

https://github.com/openstack/puppet-swift/blob/master/manifests/service.pp#L22

In the long term, we're working towards a "composable services" model upstream, which will enable completely skipping the configuration of any given service (basically we're breaking up the monolithic controller role to enable selecting the desired services from a list), but hopefully this is an acceptable interim workaround?
Comment 4 Mike Orazi 2016-03-24 14:18:38 EDT
Wanted to confirm if the env yaml documented above is a reasonable workaround for this timeframe.
Comment 5 Giulio Fidente 2016-03-24 14:40:40 EDT
Glance defaults to the Swift backend, if Swift is unavailable one should configure Glance with either Ceph/NFS or the file backend.

For example, to configure it with NFS the following should be added to the env file:

parameter_defaults:
  GlanceBackend: file
  ## Whether to make Glance 'file' backend a mount managed by Pacemaker
  GlanceFilePcmkManage: true
  ## File system type of the mount
  GlanceFilePcmkFstype: nfs
  ## Pacemaker mount point, e.g. '192.168.122.1:/export/glance' for NFS
  GlanceFilePcmkDevice: '192.168.122.1:/export/'
  ## Options for the mount managed by Pacemaker
  GlanceFilePcmkOptions: ''


We might also need to remove or change the Swift endpoint from Keystone after the deployment is finished and this has to be done after the deployment is finished, a post-deployment script won't be enough.
Comment 7 Mike Burns 2016-03-31 18:36:01 EDT
*** Bug 1322985 has been marked as a duplicate of this bug. ***
Comment 8 Alan Bishop 2016-04-01 08:38:54 EDT
The method described in comment #3 isn't working for me. The hiera data is correct:

[root@overcloud-controller-0 ~]# hiera swift::proxy::enabled
false
[root@overcloud-controller-0 ~]# hiera swift::service::enabled
false

And yet Swift is clearly enabled:

[root@overcloud-controller-0 ~]# systemctl list-units *swift*
UNIT                                         LOAD   ACTIVE SUB     DESCRIPTION
openstack-swift-account-auditor.service      loaded active running OpenStack Object Storage (swift) - Account Auditor
openstack-swift-account-reaper.service       loaded active running OpenStack Object Storage (swift) - Account Reaper
openstack-swift-account-replicator.service   loaded active running OpenStack Object Storage (swift) - Account Replicator
openstack-swift-account.service              loaded active running OpenStack Object Storage (swift) - Account Server
openstack-swift-container-auditor.service    loaded active running OpenStack Object Storage (swift) - Container Auditor
openstack-swift-container-replicator.service loaded active running OpenStack Object Storage (swift) - Container Replicator
openstack-swift-container-updater.service    loaded active running OpenStack Object Storage (swift) - Container Updater
openstack-swift-container.service            loaded active running OpenStack Object Storage (swift) - Container Server
openstack-swift-object-auditor.service       loaded active running OpenStack Object Storage (swift) - Object Auditor
openstack-swift-object-replicator.service    loaded active running OpenStack Object Storage (swift) - Object Replicator
openstack-swift-object-updater.service       loaded active running OpenStack Object Storage (swift) - Object Updater
openstack-swift-object.service               loaded active running OpenStack Object Storage (swift) - Object Server
openstack-swift-proxy.service                loaded active running OpenStack Object Storage (swift) - Proxy Server

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

13 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
[root@overcloud-controller-0 ~]# netstat -ltn | grep 8080
tcp        0      0 192.168.170.23:8080     0.0.0.0:*               LISTEN     
tcp        0      0 192.168.190.125:8080    0.0.0.0:*               LISTEN     
tcp        0      0 192.168.170.20:8080     0.0.0.0:*               LISTEN     


This is with a fresh deployment of OSP-8 Beta 9.
Comment 9 Mike Orazi 2016-04-01 10:29:04 EDT
I'd be interested to see if we can determine the value for:  

swift::proxy::manage_service

I wonder if we can also poke a value in to

swift::service::service_ensure 

( https://github.com/openstack/puppet-swift/blob/master/manifests/service.pp#L18 )

to ensure the service is stopped.
Comment 10 Alan Bishop 2016-04-01 11:42:36 EDT
I don't know how to determine those values other than seeing whether the variables are set in hiera. As I suspected, they were not set:

[root@overcloud-controller-0 hieradata]# hiera swift::proxy::manage_service
nil
[root@overcloud-controller-0 hieradata]# hiera swift::service::service_ensure
nil

So, I added them to the controllerExtraConfig's parameter_defaults, and redeployed. Unfortunately Swift is still running (same data as I showed in comment #8).

But the new items _are_ set in hiera now:

[root@overcloud-controller-0 ~]# hiera swift::proxy::manage_service
false
[root@overcloud-controller-0 ~]# hiera swift::service::service_ensure 
false
Comment 12 Alan Bishop 2016-04-04 07:31:47 EDT
OK, I figured out how to stop the service from running. The issue is overcloud_controller_pacemaker.pp wants to fiddle with the manage_service and enabled settings. Luckily, most of this is wrapped by an additional test for whether enable_swift_storage is true. So the key to getting the service to not run is:

1) Set swift::service::enabled false in hiera data
2) Set ControllerEnableSwiftStorage false in parameter_defaults

However, it seems the pacemaker always wants to enable the proxy, because this code executes regardless of whether Swift storage is enabled:

  # swift proxy
  class { '::swift::proxy' :
    manage_service => $non_pcmk_start,
    enabled        => $non_pcmk_start,
  }

[root@overcloud-controller-0 ~]# systemctl list-units *swift*
UNIT                          LOAD   ACTIVE SUB     DESCRIPTION
openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
[root@overcloud-controller-0 ~]# systemctl list-units *swift*
UNIT                          LOAD   ACTIVE SUB     DESCRIPTION
openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
[root@overcloud-controller-0 ~]# netstat -ltn | grep 8080
tcp        0      0 192.168.170.23:8080     0.0.0.0:*               LISTEN     
tcp        0      0 192.168.190.125:8080    0.0.0.0:*               LISTEN     
tcp        0      0 192.168.170.20:8080     0.0.0.0:*               LISTEN     

So the new question is, can I prevent the proxy from running?
Comment 13 Alan Bishop 2016-04-13 16:57:29 EDT
No longer a delljs5.0 blocker because we have a workaround.
Comment 15 Giulio Fidente 2016-07-08 05:40:38 EDT
This should now be possible via composable roles setting:

  OS::TripleO::Services::SwiftProxy: None
  OS::TripleO::Services::SwiftStorage: None
Comment 16 Steve Relf 2016-07-19 03:43:33 EDT
(In reply to Alan Bishop from comment #12)
> OK, I figured out how to stop the service from running. The issue is
> overcloud_controller_pacemaker.pp wants to fiddle with the manage_service
> and enabled settings. Luckily, most of this is wrapped by an additional test
> for whether enable_swift_storage is true. So the key to getting the service
> to not run is:
......
> So the new question is, can I prevent the proxy from running?

Did anyone manage to work this through. Apart form the cryptic comment from Giulio, im still at a loss as to how to disable swift using the tripleo templates.
Comment 17 Ramon Acedo 2016-07-19 12:49:43 EDT
I noticed a workaround reported:

Disable haproxy config:

cat network-environment.yaml
  controllerExtraConfig:
    swift::proxy::enabled: false
    swift::service::enabled: false
    tripleo::loadbalancer::swift_proxy_server: false

Stop swift in a bash script on post-deploy:
# stop openstack-swift services and disable them on startup
# we don't want this to throw an error and make our deployment fail, so pipe stderr to /dev/null
systemctl list-units | grep openstack-swift | awk '{print $1}' | xargs -I {} systemctl stop {} 1>/dev/null 2>&1
systemctl list-units | grep openstack-swift | awk '{print $1}' | xargs -I {} systemctl disable {} 1>/dev/null 2>&1

I haven't tested it though.

With regards to Giulio's comment #15, composable roles are available upstream at the moment but not in OSP 8. More information here: http://docs.openstack.org/developer/tripleo-docs/tht_walkthrough/tht_walkthrough.html
Comment 18 Giulio Fidente 2016-08-17 12:53:20 EDT
Given from OSPd10 this feature is given by composable roles, as per comment #15, I am re-targeting this BZ to OSPd9 to see if it can be considered for that release.
Comment 19 Mike Burns 2016-08-17 13:01:48 EDT
Features generally are not backported.  If this gets implemented as part of the composable roles feature, we should update it appropriately for testing with OSP 10.

Let's make this dependent on the composable roles feature bug 1357551
Comment 20 Giulio Fidente 2016-08-17 13:02:22 EDT
A sample environment file to use if disabling Swift (OSPd10 only):

  parameter_defaults:
    OS::TripleO::Services::SwiftProxy: OS::Heat::None
    OS::TripleO::Services::SwiftStorage: OS::Heat::None
    OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None

When needed, storage-environment.yaml can also be used to switch the CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift instead.
Comment 21 Marius Cornea 2016-08-29 06:50:18 EDT
(In reply to Giulio Fidente from comment #20)
> A sample environment file to use if disabling Swift (OSPd10 only):
> 
>   parameter_defaults:
>     OS::TripleO::Services::SwiftProxy: OS::Heat::None
>     OS::TripleO::Services::SwiftStorage: OS::Heat::None
>     OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None

Just a quick note - s/parameter_defaults/resource_registry/ in the environment file:

resource_registry:
  OS::TripleO::Services::SwiftProxy: OS::Heat::None
  OS::TripleO::Services::SwiftStorage: OS::Heat::None
  OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None

> When needed, storage-environment.yaml can also be used to switch the
> CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift
> instead.
Comment 22 Marius Cornea 2016-08-29 07:27:55 EDT
(In reply to Marius Cornea from comment #21)
> (In reply to Giulio Fidente from comment #20)
> > A sample environment file to use if disabling Swift (OSPd10 only):
> > 
> >   parameter_defaults:
> >     OS::TripleO::Services::SwiftProxy: OS::Heat::None
> >     OS::TripleO::Services::SwiftStorage: OS::Heat::None
> >     OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None
> 
> Just a quick note - s/parameter_defaults/resource_registry/ in the
> environment file:
> 
> resource_registry:
>   OS::TripleO::Services::SwiftProxy: OS::Heat::None
>   OS::TripleO::Services::SwiftStorage: OS::Heat::None
>   OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None
> 
> > When needed, storage-environment.yaml can also be used to switch the
> > CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift
> > instead.

I tested this and Swift didn't get configured on the overcloud controllers but I see 2 remaining bits:

1. The Swift endpoints still gets created:

[stack@undercloud ~]$ . overcloudrc 
[stack@undercloud ~]$ openstack catalog show swift
+-----------+--------------------------------------------------------------------------------+
| Field     | Value                                                                          |
+-----------+--------------------------------------------------------------------------------+
| endpoints | regionOne                                                                      |
|           |   publicURL: http://172.16.18.26:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec |
|           |   internalURL: http://10.0.0.143:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec |
|           |   adminURL: http://10.0.0.143:8080/v1                                          |
|           |                                                                                |
| name      | swift                                                                          |
| type      | object-store                                                                   |
+-----------+--------------------------------------------------------------------------------+

2. HAProxy still configures the swift_proxy_server section:

[root@overcloud-controller-0 ~]# grep -A5 swift /etc/haproxy/haproxy.cfg 
listen swift_proxy_server
  bind 10.0.0.143:8080 transparent
  bind 172.16.18.26:8080 transparent
  server overcloud-controller-0 10.0.0.138:8080 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 10.0.0.141:8080 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 10.0.0.139:8080 check fall 5 inter 2000 rise 2


Are there any steps that we can use to avoid creating those?

Thanks!
Comment 23 Marius Cornea 2016-08-29 07:55:50 EDT
(In reply to Marius Cornea from comment #22)
> (In reply to Marius Cornea from comment #21)
> > (In reply to Giulio Fidente from comment #20)
> > > A sample environment file to use if disabling Swift (OSPd10 only):
> > > 
> > >   parameter_defaults:
> > >     OS::TripleO::Services::SwiftProxy: OS::Heat::None
> > >     OS::TripleO::Services::SwiftStorage: OS::Heat::None
> > >     OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None
> > 
> > Just a quick note - s/parameter_defaults/resource_registry/ in the
> > environment file:
> > 
> > resource_registry:
> >   OS::TripleO::Services::SwiftProxy: OS::Heat::None
> >   OS::TripleO::Services::SwiftStorage: OS::Heat::None
> >   OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None
> > 
> > > When needed, storage-environment.yaml can also be used to switch the
> > > CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift
> > > instead.
> 
> I tested this and Swift didn't get configured on the overcloud controllers
> but I see 2 remaining bits:
> 
> 1. The Swift endpoints still gets created:
> 
> [stack@undercloud ~]$ . overcloudrc 
> [stack@undercloud ~]$ openstack catalog show swift
> +-----------+----------------------------------------------------------------
> ----------------+
> | Field     | Value                                                         
> |
> +-----------+----------------------------------------------------------------
> ----------------+
> | endpoints | regionOne                                                     
> |
> |           |   publicURL:
> http://172.16.18.26:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec |
> |           |   internalURL:
> http://10.0.0.143:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec |
> |           |   adminURL: http://10.0.0.143:8080/v1                         
> |
> |           |                                                               
> |
> | name      | swift                                                         
> |
> | type      | object-store                                                  
> |
> +-----------+----------------------------------------------------------------
> ----------------+
> 
> 2. HAProxy still configures the swift_proxy_server section:
> 
> [root@overcloud-controller-0 ~]# grep -A5 swift /etc/haproxy/haproxy.cfg 
> listen swift_proxy_server
>   bind 10.0.0.143:8080 transparent
>   bind 172.16.18.26:8080 transparent
>   server overcloud-controller-0 10.0.0.138:8080 check fall 5 inter 2000 rise
> 2
>   server overcloud-controller-1 10.0.0.141:8080 check fall 5 inter 2000 rise
> 2
>   server overcloud-controller-2 10.0.0.139:8080 check fall 5 inter 2000 rise
> 2

This worked to avoid creating the HAproxy config for Swift proxy:

parameter_defaults:
  ControllerExtraConfig:
    tripleo::haproxy::swift_proxy_server: false

> 
> Are there any steps that we can use to avoid creating those?
> 
> Thanks!
Comment 24 Giulio Fidente 2016-08-29 08:33:08 EDT
hi Marius, thanks for the amends in comments #21 and #23, we should probably sum those up in a single bug doc text.

The remaining bits to avoid creation of the endpoint will be coming with https://review.openstack.org/#/c/353307/, I moved the BZ on POST a bit too early, the bulk of the changes come with composable roles ... but I forgot about the endpoints.

We should probably re-test this when that change is available in THT as well. I am linking it here and moving back to ON_DEV.
Comment 25 arkady kanevsky 2016-08-29 09:58:15 EDT
Marius comment brought a question.
Do we need 2 proxy when we have swift using Ceph?
One for swift proxy and one for Ceph proxy, both on controller nodes?

Do we have a way to configure swift to use local ceph endpoint on a controller node? That will handle controller node failure but not services failure.

While double proxy is more resilient and allows to extend RGW cluster outside controller nodes the performance impact is not known.

Giulio,
where is code that take advantage of keystone check in https://review.openstack.org/#/c/353307/?
Comment 26 Giulio Fidente 2016-08-29 12:31:21 EDT
hi Arkady,

(In reply to arkady kanevsky from comment #25)
> Marius comment brought a question.
> Do we need 2 proxy when we have swift using Ceph?
> One for swift proxy and one for Ceph proxy, both on controller nodes?

yes we do and this is why disabling the roles does not disable proxying as well ... unless one passes tripleo::haproxy::swift_proxy_server: false
 
> Do we have a way to configure swift to use local ceph endpoint on a
> controller node? That will handle controller node failure but not services
> failure.

I am not sure I got the question right, maybe we can sync up on IRC (#tripleo)?
 
> While double proxy is more resilient and allows to extend RGW cluster
> outside controller nodes the performance impact is not known.
> 
> Giulio,
> where is code that take advantage of keystone check in
> https://review.openstack.org/#/c/353307/

Looks like it's easier to sync up on the above on IRC as well, we might want additional/different BZs (or RFEs) for the other functionalities.
Comment 27 Giulio Fidente 2016-08-29 12:32:41 EDT
(In reply to Giulio Fidente from comment #26)
> hi Arkady,
> 
> (In reply to arkady kanevsky from comment #25)
> > Marius comment brought a question.
> > Do we need 2 proxy when we have swift using Ceph?
> > One for swift proxy and one for Ceph proxy, both on controller nodes?
> 
> yes we do and this is why disabling the roles does not disable proxying as
> well ... unless one passes tripleo::haproxy::swift_proxy_server: false

so a better answer here would be, no we don't need two haproxy stanzas, we can use the same stanza which swift uses for ceph/rgw ... unless we don't have any of the two, in which case we can disable it via the mentioned boolean
Comment 28 Marius Cornea 2016-08-30 04:29:55 EDT
https://review.openstack.org/#/c/359645/7 conditionally exposes the haproxy endpoints so after it lands we shouldn't need to use the ControllerExtraConfig to disable swift_proxy_server
Comment 29 Giulio Fidente 2016-09-29 12:21:45 EDT
All needed patches were merged upstream, *today* to test this the only needed thing should be to use an environment file with the following:

  resource_registry:
    OS::TripleO::Services::SwiftProxy: OS::Heat::None
    OS::TripleO::Services::SwiftStorage: OS::Heat::None
    OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None
Comment 37 Alan Bishop 2016-12-13 08:11:22 EST
Regarding the Doc Text, is it worth mentioning the environment file that enables the Ceph RGW (ceph-radosgw.yaml) includes these resources in order to ensure Swift is disabled?
Comment 38 Giulio Fidente 2016-12-14 04:39:28 EST
(In reply to Alan Bishop from comment #37)
> Regarding the Doc Text, is it worth mentioning the environment file that
> enables the Ceph RGW (ceph-radosgw.yaml) includes these resources in order
> to ensure Swift is disabled?

hi Alan,

that environment file will also deploy RGW as a replacement for Swift so I think it'd be best to document that in a RGW related BZ, probably BZ 1309460.
Comment 40 Alan Bishop 2016-12-14 09:09:58 EST
Hi Giulio,

My concern was readers who deploy the Ceph RGW might think that they also needed to disable Swift. But I see the doc text you added to BZ 1309460 and that addresses my concern. Thanks!

But I have a new concern regarding Deepti's doc text, and that is the implication that Swift is the only component that supplies object service in the overcloud. If the Ceph RGW is deployed then the overcloud will have an object service, including the endpoint. I hope I'm not splitting hairs, but my recommendation is the text indicate the resources will disable the "Swift Object Storage" and not simply "Object Storage." That is, add "Swift" as a prefix.
Comment 41 errata-xmlrpc 2016-12-14 10:21:31 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html

Note You need to log in before you can comment on or make changes to this bug.