Description of problem: overcloud swift commands failing with 503 error Version-Release number of selected component (if applicable): (overcloud) [stack@undercloud-0 ~]$ rpm -qa | grep -i swift puppet-swift-15.4.1-0.20191014152413.654e964.el8ost.noarch python3-swiftclient-3.8.1-0.20190920094747.72b90fe.el8ost.noarch (overcloud) [stack@undercloud-0 ~]$ How reproducible: always Steps to Reproduce: 1. . ./overcloudrc 2. swift list, swift stat, etc. 3. Actual results: overcloud) [stack@undercloud-0 ~]$ swift list Account GET failed: http://10.0.0.114:8080/v1/AUTH_82fde0df75c646c6b5efb3c51563cc78?format=json 503 Service Unavailable [first 60 chars of response] b'<html><body><h1>503 Service Unavailable</h1>\nNo server is av' (overcloud) [stack@undercloud-0 ~]$ swift stat -v Account HEAD failed: http://10.0.0.114:8080/v1/AUTH_82fde0df75c646c6b5efb3c51563cc78 503 Service Unavailable (overcloud) [stack@undercloud-0 ~]$ Expected results: commands should succeed Additional info: (overcloud) [stack@undercloud-0 ~]$ . ./stackrc (undercloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 1579b2d5-40a1-4c61-80c2-9b29516f63e5 | controller-0 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | controller | | d3ad5e2d-9e25-4571-88a2-db532b236c5d | controller-2 | ACTIVE | ctlplane=192.168.24.33 | overcloud-full | controller | | 77ad5262-757f-4990-a3b4-fc5b530a0e13 | controller-1 | ACTIVE | ctlplane=192.168.24.13 | overcloud-full | controller | | f6e23c69-66a0-4f97-85c6-490d210cbe87 | compute-0 | ACTIVE | ctlplane=192.168.24.40 | overcloud-full | compute | | 80f7aad2-b811-4816-820f-fd595257ddeb | ceph-0 | ACTIVE | ctlplane=192.168.24.53 | overcloud-full | ceph | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ (undercloud) [stack@undercloud-0 ~]$ ssh -t heat-admin.24.20 Warning: Permanently added '192.168.24.20' (ECDSA) to the list of known hosts. This system is not registered to Red Hat Insights. See https://cloud.redhat.com/ To register this system, run: insights-client --register [heat-admin@controller-0 ~]$ sudo su - Last login: Wed Dec 25 05:35:43 UTC 2019 [root@controller-0 ~]# podman ps | grep swift 13b3362e5580 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-proxy-server:20191217.1 dumb-init --singl... 3 days ago Up 6 seconds ago swift_proxy 817b0ab4221d undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-object:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_rsync 1ef84b8aa177 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-object:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_object_updater 8d4b5937f6c6 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-object:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_object_server d27ed6b01278 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-object:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_object_replicator 1d62f2e35aa7 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-proxy-server:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_object_expirer 3a7c4528f7f4 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-object:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_object_auditor 296ba2f9a8b4 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-container:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_container_updater 43c72aeb4acd undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-container:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_container_server 593cc012dd53 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-container:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_container_replicator b47fa105a591 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-container:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_container_auditor f8c7dfe38bc9 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-account:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_account_server ad12837d2342 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-account:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_account_replicator 7bc3191a5d14 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-account:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_account_reaper c2ba8a1b4f17 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-account:20191217.1 dumb-init --singl... 3 days ago Up 3 days ago swift_account_auditor [root@controller-0 ~]# --- [root@controller-0 ~]# podman logs swift_proxy | grep ERROR ... 2019-12-25 12:12:38.651 8 ERROR castellan return uuid.UUID(ref_pieces[-1]) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib64/python3.6/uuid.py", line 140, in __init__ 2019-12-25 12:12:38.651 8 ERROR castellan raise ValueError('badly formed hexadecimal UUID string') 2019-12-25 12:12:38.651 8 ERROR castellan ValueError: badly formed hexadecimal UUID string 2019-12-25 12:12:38.651 8 ERROR castellan 2019-12-25 12:12:38.651 8 ERROR castellan During handling of the above exception, another exception occurred: 2019-12-25 12:12:38.651 8 ERROR castellan 2019-12-25 12:12:38.651 8 ERROR castellan Traceback (most recent call last): 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/bin/swift-proxy-server", line 23, in <module> 2019-12-25 12:12:38.651 8 ERROR castellan sys.exit(run_wsgi(conf_file, 'proxy-server', **options)) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/swift/common/wsgi.py", line 1114, in run_wsgi 2019-12-25 12:12:38.651 8 ERROR castellan loadapp(conf_path, global_conf=global_conf) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/swift/common/wsgi.py", line 400, in loadapp 2019-12-25 12:12:38.651 8 ERROR castellan return ctx.create() 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 710, in create 2019-12-25 12:12:38.651 8 ERROR castellan return self.object_type.invoke(self) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/paste/deploy/loadwsgi.py", line 207, in invoke 2019-12-25 12:12:38.651 8 ERROR castellan app = filter(app) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/swift/common/middleware/crypto/kms_keymaster.py", line 111, in kms_keymaster_filter 2019-12-25 12:12:38.651 8 ERROR castellan return KmsKeyMaster(app, conf) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/swift/common/middleware/crypto/keymaster.py", line 210, in __init__ 2019-12-25 12:12:38.651 8 ERROR castellan self._root_secrets = self._get_root_secret(conf) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/swift/common/middleware/crypto/kms_keymaster.py", line 78, in _get_root_secret 2019-12-25 12:12:38.651 8 ERROR castellan key = manager.get(ctxt, key_id) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 563, in get 2019-12-25 12:12:38.651 8 ERROR castellan secret = self._get_secret(context, managed_object_id) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py", line 537, in _get_secret 2019-12-25 12:12:38.651 8 ERROR castellan return barbican_client.secrets.get(secret_ref) 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/barbicanclient/v1/secrets.py", line 459, in get 2019-12-25 12:12:38.651 8 ERROR castellan base.validate_ref_and_return_uuid(secret_ref, 'Secret') 2019-12-25 12:12:38.651 8 ERROR castellan File "/usr/lib/python3.6/site-packages/barbicanclient/base.py", line 48, in validate_ref_and_return_uuid 2019-12-25 12:12:38.651 8 ERROR castellan raise ValueError('{0} incorrectly specified.'.format(entity)) 2019-12-25 12:12:38.651 8 ERROR castellan ValueError: Secret incorrectly specified. 2019-12-25 12:12:38.651 8 ERROR castellan [root@controller-0 ~]# date Wed Dec 25 12:13:54 UTC 2019 [root@controller-0 ~]# ^^^ barbican errors in container swift_proxy no glaring errors in /var/log/containers/swift
confirmed without barbican swift is indeed functional: [stack@undercloud-0 ~]$ . ./overcloudrc (overcloud) [stack@undercloud-0 ~]$ rhos-release -L Installed repositories (rhel-8.1): 16 ceph-4 ceph-osd-4 rhel-8.1 (overcloud) [stack@undercloud-0 ~]$ swift stat -v Invalid -W option ignored: invalid action: '"ignore' StorageURL: http://10.0.0.134:8080/v1/AUTH_5fb2626f397b48b4b2157aac66a8fb81 Auth Token: gAAAAABeFBS1rqWQjgBny216CfkN5yM_0dQEl6Gc7-_-bdW_x3SAuSxBv-xH7iMrbCqDQyNM2fiIwhfElPuhcKz5bTBRhAN4KXeglltelcAQ1JAdShHtpeTLwkFx6xnM_fiGNv0_a6YVH-7mXlbTWLlhBfzhtjrUuammE-nX2Ih7hAGc0QXIGFA Account: AUTH_5fb2626f397b48b4b2157aac66a8fb81 Containers: 0 Objects: 0 Bytes: 0 Content-Type: text/plain; charset=utf-8 X-Timestamp: 1578374326.50316 X-Put-Timestamp: 1578374326.50316 X-Trans-Id: tx63f250fc28a9447b947f1-005e1414b5 X-Openstack-Request-Id: tx63f250fc28a9447b947f1-005e1414b5 (overcloud) [stack@undercloud-0 ~]$ --- (overcloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock1.rdu2.redhat.com \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/network/dvr-override.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/nodes_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e /home/stack/virt/extra_templates.yaml \ --log-file overcloud_deployment_5.log (overcloud) [stack@undercloud-0 ~]$
Ade and Douglas investigated this (thx!), looks like the key ID is not properly set in keymaster.conf. All other parameters are set, the script itself is executed. However, it's a permission issue it seems: [root@controller-0 stdouts]# cat /var/log/containers/stdouts/set_swift_secret.log.1 2020-01-02T14:04:22.032696455+00:00 stdout F retrieve key_id 2020-01-02T14:04:27.350469375+00:00 stdout F set key_id in keymaster.conf 2020-01-02T14:04:34.100754097+00:00 stderr F [Errno 13] Permission denied: '/etc/swift/keymaster.conf.sqbo9aej.tmp'
Opened an upstream bug with more details (https://bugs.launchpad.net/tripleo/+bug/1858845) and submitted a fix on Gerrit (https://review.opendev.org/#/c/701600/).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0283