Description of problem: Today the default deployment of OSP 7 deploys Swift Proxy on the controller nodes as well as loopback devices for storage on the controllers. For customers who do not want object storage, there is no way to disable Swift. At best you can disable deploying the loopback devices on the controllers. However, this will just result in a Swift proxy with no backend Version-Release number of selected component (if applicable): RHEL OSP 7 (all versions) How reproducible: This can be reproduced by doing a default deployment of the overcloud from Director. Expected results: An 'enable swift' flag should be given in the controller that allows you to disable deployment of Swift proxy. Additional info:
It should already be possible to disable the services directly via puppet ExtraConfig overrides, something like this (untested atm): $ cat disable_swift_env.yaml parameter_defaults: controllerExtraConfig: swift::proxy::enabled: false swift::service::enabled: false openstack overcloud deploy --templates -e disable_swift_env.yaml Basically this wires in a boolean directly to the puppet hieradata, which will ovetrride the defaults here: https://github.com/openstack/puppet-swift/blob/master/manifests/proxy.pp#L85 https://github.com/openstack/puppet-swift/blob/master/manifests/service.pp#L22 In the long term, we're working towards a "composable services" model upstream, which will enable completely skipping the configuration of any given service (basically we're breaking up the monolithic controller role to enable selecting the desired services from a list), but hopefully this is an acceptable interim workaround?
Wanted to confirm if the env yaml documented above is a reasonable workaround for this timeframe.
Glance defaults to the Swift backend, if Swift is unavailable one should configure Glance with either Ceph/NFS or the file backend. For example, to configure it with NFS the following should be added to the env file: parameter_defaults: GlanceBackend: file ## Whether to make Glance 'file' backend a mount managed by Pacemaker GlanceFilePcmkManage: true ## File system type of the mount GlanceFilePcmkFstype: nfs ## Pacemaker mount point, e.g. '192.168.122.1:/export/glance' for NFS GlanceFilePcmkDevice: '192.168.122.1:/export/' ## Options for the mount managed by Pacemaker GlanceFilePcmkOptions: '' We might also need to remove or change the Swift endpoint from Keystone after the deployment is finished and this has to be done after the deployment is finished, a post-deployment script won't be enough.
*** Bug 1322985 has been marked as a duplicate of this bug. ***
The method described in comment #3 isn't working for me. The hiera data is correct: [root@overcloud-controller-0 ~]# hiera swift::proxy::enabled false [root@overcloud-controller-0 ~]# hiera swift::service::enabled false And yet Swift is clearly enabled: [root@overcloud-controller-0 ~]# systemctl list-units *swift* UNIT LOAD ACTIVE SUB DESCRIPTION openstack-swift-account-auditor.service loaded active running OpenStack Object Storage (swift) - Account Auditor openstack-swift-account-reaper.service loaded active running OpenStack Object Storage (swift) - Account Reaper openstack-swift-account-replicator.service loaded active running OpenStack Object Storage (swift) - Account Replicator openstack-swift-account.service loaded active running OpenStack Object Storage (swift) - Account Server openstack-swift-container-auditor.service loaded active running OpenStack Object Storage (swift) - Container Auditor openstack-swift-container-replicator.service loaded active running OpenStack Object Storage (swift) - Container Replicator openstack-swift-container-updater.service loaded active running OpenStack Object Storage (swift) - Container Updater openstack-swift-container.service loaded active running OpenStack Object Storage (swift) - Container Server openstack-swift-object-auditor.service loaded active running OpenStack Object Storage (swift) - Object Auditor openstack-swift-object-replicator.service loaded active running OpenStack Object Storage (swift) - Object Replicator openstack-swift-object-updater.service loaded active running OpenStack Object Storage (swift) - Object Updater openstack-swift-object.service loaded active running OpenStack Object Storage (swift) - Object Server openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 13 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. [root@overcloud-controller-0 ~]# netstat -ltn | grep 8080 tcp 0 0 192.168.170.23:8080 0.0.0.0:* LISTEN tcp 0 0 192.168.190.125:8080 0.0.0.0:* LISTEN tcp 0 0 192.168.170.20:8080 0.0.0.0:* LISTEN This is with a fresh deployment of OSP-8 Beta 9.
I'd be interested to see if we can determine the value for: swift::proxy::manage_service I wonder if we can also poke a value in to swift::service::service_ensure ( https://github.com/openstack/puppet-swift/blob/master/manifests/service.pp#L18 ) to ensure the service is stopped.
I don't know how to determine those values other than seeing whether the variables are set in hiera. As I suspected, they were not set: [root@overcloud-controller-0 hieradata]# hiera swift::proxy::manage_service nil [root@overcloud-controller-0 hieradata]# hiera swift::service::service_ensure nil So, I added them to the controllerExtraConfig's parameter_defaults, and redeployed. Unfortunately Swift is still running (same data as I showed in comment #8). But the new items _are_ set in hiera now: [root@overcloud-controller-0 ~]# hiera swift::proxy::manage_service false [root@overcloud-controller-0 ~]# hiera swift::service::service_ensure false
OK, I figured out how to stop the service from running. The issue is overcloud_controller_pacemaker.pp wants to fiddle with the manage_service and enabled settings. Luckily, most of this is wrapped by an additional test for whether enable_swift_storage is true. So the key to getting the service to not run is: 1) Set swift::service::enabled false in hiera data 2) Set ControllerEnableSwiftStorage false in parameter_defaults However, it seems the pacemaker always wants to enable the proxy, because this code executes regardless of whether Swift storage is enabled: # swift proxy class { '::swift::proxy' : manage_service => $non_pcmk_start, enabled => $non_pcmk_start, } [root@overcloud-controller-0 ~]# systemctl list-units *swift* UNIT LOAD ACTIVE SUB DESCRIPTION openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. [root@overcloud-controller-0 ~]# systemctl list-units *swift* UNIT LOAD ACTIVE SUB DESCRIPTION openstack-swift-proxy.service loaded active running OpenStack Object Storage (swift) - Proxy Server LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. [root@overcloud-controller-0 ~]# netstat -ltn | grep 8080 tcp 0 0 192.168.170.23:8080 0.0.0.0:* LISTEN tcp 0 0 192.168.190.125:8080 0.0.0.0:* LISTEN tcp 0 0 192.168.170.20:8080 0.0.0.0:* LISTEN So the new question is, can I prevent the proxy from running?
No longer a delljs5.0 blocker because we have a workaround.
This should now be possible via composable roles setting: OS::TripleO::Services::SwiftProxy: None OS::TripleO::Services::SwiftStorage: None
(In reply to Alan Bishop from comment #12) > OK, I figured out how to stop the service from running. The issue is > overcloud_controller_pacemaker.pp wants to fiddle with the manage_service > and enabled settings. Luckily, most of this is wrapped by an additional test > for whether enable_swift_storage is true. So the key to getting the service > to not run is: ...... > So the new question is, can I prevent the proxy from running? Did anyone manage to work this through. Apart form the cryptic comment from Giulio, im still at a loss as to how to disable swift using the tripleo templates.
I noticed a workaround reported: Disable haproxy config: cat network-environment.yaml controllerExtraConfig: swift::proxy::enabled: false swift::service::enabled: false tripleo::loadbalancer::swift_proxy_server: false Stop swift in a bash script on post-deploy: # stop openstack-swift services and disable them on startup # we don't want this to throw an error and make our deployment fail, so pipe stderr to /dev/null systemctl list-units | grep openstack-swift | awk '{print $1}' | xargs -I {} systemctl stop {} 1>/dev/null 2>&1 systemctl list-units | grep openstack-swift | awk '{print $1}' | xargs -I {} systemctl disable {} 1>/dev/null 2>&1 I haven't tested it though. With regards to Giulio's comment #15, composable roles are available upstream at the moment but not in OSP 8. More information here: http://docs.openstack.org/developer/tripleo-docs/tht_walkthrough/tht_walkthrough.html
Given from OSPd10 this feature is given by composable roles, as per comment #15, I am re-targeting this BZ to OSPd9 to see if it can be considered for that release.
Features generally are not backported. If this gets implemented as part of the composable roles feature, we should update it appropriately for testing with OSP 10. Let's make this dependent on the composable roles feature bug 1357551
A sample environment file to use if disabling Swift (OSPd10 only): parameter_defaults: OS::TripleO::Services::SwiftProxy: OS::Heat::None OS::TripleO::Services::SwiftStorage: OS::Heat::None OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None When needed, storage-environment.yaml can also be used to switch the CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift instead.
(In reply to Giulio Fidente from comment #20) > A sample environment file to use if disabling Swift (OSPd10 only): > > parameter_defaults: > OS::TripleO::Services::SwiftProxy: OS::Heat::None > OS::TripleO::Services::SwiftStorage: OS::Heat::None > OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None Just a quick note - s/parameter_defaults/resource_registry/ in the environment file: resource_registry: OS::TripleO::Services::SwiftProxy: OS::Heat::None OS::TripleO::Services::SwiftStorage: OS::Heat::None OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None > When needed, storage-environment.yaml can also be used to switch the > CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift > instead.
(In reply to Marius Cornea from comment #21) > (In reply to Giulio Fidente from comment #20) > > A sample environment file to use if disabling Swift (OSPd10 only): > > > > parameter_defaults: > > OS::TripleO::Services::SwiftProxy: OS::Heat::None > > OS::TripleO::Services::SwiftStorage: OS::Heat::None > > OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None > > Just a quick note - s/parameter_defaults/resource_registry/ in the > environment file: > > resource_registry: > OS::TripleO::Services::SwiftProxy: OS::Heat::None > OS::TripleO::Services::SwiftStorage: OS::Heat::None > OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None > > > When needed, storage-environment.yaml can also be used to switch the > > CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift > > instead. I tested this and Swift didn't get configured on the overcloud controllers but I see 2 remaining bits: 1. The Swift endpoints still gets created: [stack@undercloud ~]$ . overcloudrc [stack@undercloud ~]$ openstack catalog show swift +-----------+--------------------------------------------------------------------------------+ | Field | Value | +-----------+--------------------------------------------------------------------------------+ | endpoints | regionOne | | | publicURL: http://172.16.18.26:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec | | | internalURL: http://10.0.0.143:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec | | | adminURL: http://10.0.0.143:8080/v1 | | | | | name | swift | | type | object-store | +-----------+--------------------------------------------------------------------------------+ 2. HAProxy still configures the swift_proxy_server section: [root@overcloud-controller-0 ~]# grep -A5 swift /etc/haproxy/haproxy.cfg listen swift_proxy_server bind 10.0.0.143:8080 transparent bind 172.16.18.26:8080 transparent server overcloud-controller-0 10.0.0.138:8080 check fall 5 inter 2000 rise 2 server overcloud-controller-1 10.0.0.141:8080 check fall 5 inter 2000 rise 2 server overcloud-controller-2 10.0.0.139:8080 check fall 5 inter 2000 rise 2 Are there any steps that we can use to avoid creating those? Thanks!
(In reply to Marius Cornea from comment #22) > (In reply to Marius Cornea from comment #21) > > (In reply to Giulio Fidente from comment #20) > > > A sample environment file to use if disabling Swift (OSPd10 only): > > > > > > parameter_defaults: > > > OS::TripleO::Services::SwiftProxy: OS::Heat::None > > > OS::TripleO::Services::SwiftStorage: OS::Heat::None > > > OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None > > > > Just a quick note - s/parameter_defaults/resource_registry/ in the > > environment file: > > > > resource_registry: > > OS::TripleO::Services::SwiftProxy: OS::Heat::None > > OS::TripleO::Services::SwiftStorage: OS::Heat::None > > OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None > > > > > When needed, storage-environment.yaml can also be used to switch the > > > CinderBackup, Glance and Gnocchi backend to Ceph; these default to Swift > > > instead. > > I tested this and Swift didn't get configured on the overcloud controllers > but I see 2 remaining bits: > > 1. The Swift endpoints still gets created: > > [stack@undercloud ~]$ . overcloudrc > [stack@undercloud ~]$ openstack catalog show swift > +-----------+---------------------------------------------------------------- > ----------------+ > | Field | Value > | > +-----------+---------------------------------------------------------------- > ----------------+ > | endpoints | regionOne > | > | | publicURL: > http://172.16.18.26:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec | > | | internalURL: > http://10.0.0.143:8080/v1/AUTH_aa788b99c19348689a32bdc9a8e62aec | > | | adminURL: http://10.0.0.143:8080/v1 > | > | | > | > | name | swift > | > | type | object-store > | > +-----------+---------------------------------------------------------------- > ----------------+ > > 2. HAProxy still configures the swift_proxy_server section: > > [root@overcloud-controller-0 ~]# grep -A5 swift /etc/haproxy/haproxy.cfg > listen swift_proxy_server > bind 10.0.0.143:8080 transparent > bind 172.16.18.26:8080 transparent > server overcloud-controller-0 10.0.0.138:8080 check fall 5 inter 2000 rise > 2 > server overcloud-controller-1 10.0.0.141:8080 check fall 5 inter 2000 rise > 2 > server overcloud-controller-2 10.0.0.139:8080 check fall 5 inter 2000 rise > 2 This worked to avoid creating the HAproxy config for Swift proxy: parameter_defaults: ControllerExtraConfig: tripleo::haproxy::swift_proxy_server: false > > Are there any steps that we can use to avoid creating those? > > Thanks!
hi Marius, thanks for the amends in comments #21 and #23, we should probably sum those up in a single bug doc text. The remaining bits to avoid creation of the endpoint will be coming with https://review.openstack.org/#/c/353307/, I moved the BZ on POST a bit too early, the bulk of the changes come with composable roles ... but I forgot about the endpoints. We should probably re-test this when that change is available in THT as well. I am linking it here and moving back to ON_DEV.
Marius comment brought a question. Do we need 2 proxy when we have swift using Ceph? One for swift proxy and one for Ceph proxy, both on controller nodes? Do we have a way to configure swift to use local ceph endpoint on a controller node? That will handle controller node failure but not services failure. While double proxy is more resilient and allows to extend RGW cluster outside controller nodes the performance impact is not known. Giulio, where is code that take advantage of keystone check in https://review.openstack.org/#/c/353307/?
hi Arkady, (In reply to arkady kanevsky from comment #25) > Marius comment brought a question. > Do we need 2 proxy when we have swift using Ceph? > One for swift proxy and one for Ceph proxy, both on controller nodes? yes we do and this is why disabling the roles does not disable proxying as well ... unless one passes tripleo::haproxy::swift_proxy_server: false > Do we have a way to configure swift to use local ceph endpoint on a > controller node? That will handle controller node failure but not services > failure. I am not sure I got the question right, maybe we can sync up on IRC (#tripleo)? > While double proxy is more resilient and allows to extend RGW cluster > outside controller nodes the performance impact is not known. > > Giulio, > where is code that take advantage of keystone check in > https://review.openstack.org/#/c/353307/ Looks like it's easier to sync up on the above on IRC as well, we might want additional/different BZs (or RFEs) for the other functionalities.
(In reply to Giulio Fidente from comment #26) > hi Arkady, > > (In reply to arkady kanevsky from comment #25) > > Marius comment brought a question. > > Do we need 2 proxy when we have swift using Ceph? > > One for swift proxy and one for Ceph proxy, both on controller nodes? > > yes we do and this is why disabling the roles does not disable proxying as > well ... unless one passes tripleo::haproxy::swift_proxy_server: false so a better answer here would be, no we don't need two haproxy stanzas, we can use the same stanza which swift uses for ceph/rgw ... unless we don't have any of the two, in which case we can disable it via the mentioned boolean
https://review.openstack.org/#/c/359645/7 conditionally exposes the haproxy endpoints so after it lands we shouldn't need to use the ControllerExtraConfig to disable swift_proxy_server
All needed patches were merged upstream, *today* to test this the only needed thing should be to use an environment file with the following: resource_registry: OS::TripleO::Services::SwiftProxy: OS::Heat::None OS::TripleO::Services::SwiftStorage: OS::Heat::None OS::TripleO::Services::SwiftRingBuilder: OS::Heat::None
Regarding the Doc Text, is it worth mentioning the environment file that enables the Ceph RGW (ceph-radosgw.yaml) includes these resources in order to ensure Swift is disabled?
(In reply to Alan Bishop from comment #37) > Regarding the Doc Text, is it worth mentioning the environment file that > enables the Ceph RGW (ceph-radosgw.yaml) includes these resources in order > to ensure Swift is disabled? hi Alan, that environment file will also deploy RGW as a replacement for Swift so I think it'd be best to document that in a RGW related BZ, probably BZ 1309460.
Hi Giulio, My concern was readers who deploy the Ceph RGW might think that they also needed to disable Swift. But I see the doc text you added to BZ 1309460 and that addresses my concern. Thanks! But I have a new concern regarding Deepti's doc text, and that is the implication that Swift is the only component that supplies object service in the overcloud. If the Ceph RGW is deployed then the overcloud will have an object service, including the endpoint. I hope I'm not splitting hairs, but my recommendation is the text indicate the resources will disable the "Swift Object Storage" and not simply "Object Storage." That is, add "Swift" as a prefix.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html