Created attachment 1457061 [details] Undercloud SOS report Description of problem: In the step where the Controllers are being upgraded, the process fails and exits. Version-Release number of selected component (if applicable): 10->13 The issue looks like it is around the upgrade of gnocchi. u'NO MORE HOSTS LEFT *************************************************************', u'', u'PLAY RECAP *********************************************************************', u'172.16.0.101 : ok=167 changed=45 unreachable=0 failed=1 ', u'172.16.0.105 : ok=165 changed=44 unreachable=0 failed=0 ', u'172.16.0.108 : ok=165 changed=44 unreachable=0 failed=0 ', u''] Update failed with: {u'status': u'FAILED', u'execution': {u'name': u'tripleo.package_update.v1.update_nodes', u'created_at': u'2018-07-06 16:20:03', u'updated_at': u'2018-07-06 16:20:03', u'spec': {u'tasks': {u'node_update': {u'name': u'node_update', u'on-error': u'node_update_failed', u'on-success': [{u'node_update_passed': u'<% task().result.returncode = 0 %>'}, {u'node_update_failed': u'<% task().result.returncode != 0 %>'}], u'publish': {u'output': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.ansible-playbook', u'input': {u'remote_user': u'<% $.node_user %>', u'become_user': u'root', u'ssh_private_key': u'<% $.private_key %>', u'verbosity': u'<% $.verbosity %>', u'queue_name': u'<% $.ansible_queue_name %>', u'extra_env_variables': u'<% $.ansible_extra_env_variables %>', u'skip_tags': u'<% $.skip_tags %>', u'inventory': u'<% $.inventory_file %>', u'execution_id': u'<% execution().id %>', u'module_path': u'<% $.module_path %>', u'become': True, u'trash_output': True, u'limit_hosts': u'<% $.nodes %>', u'playbook': u'<% $.work_dir %>/<% execution().id %>/<% $.playbook %>'}, u'type': u'direct'}, u'get_private_key': {u'name': u'get_private_key', u'on-success': u'node_update', u'publish': {u'private_key': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.validations.get_privkey', u'type': u'direct'}, u'node_update_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_failed', u'publish': {u'status': u'FAILED', u'message': u'Failed to update nodes - <% $.nodes %>, please see the logs.'}, u'on-success': u'notify_zaqar'}, u'node_update_passed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_passed', u'publish': {u'status': u'SUCCESS', u'message': u'Updated nodes - <% $.nodes %>'}, u'on-success': u'notify_zaqar'}, u'notify_zaqar': {u'retry': u'count=5 delay=1', u'name': u'notify_zaqar', u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}], u'version': u'2.0', u'action': u'zaqar.queue_post', u'input': {u'queue_name': u'<% $.ansible_queue_name %>', u'messages': {u'body': {u'type': u'tripleo.package_update.v1.update_nodes', u'payload': {u'status': u'<% $.status %>', u'execution': u'<% execution() %>'}}}}, u'type': u'direct'}, u'download_config': {u'name': u'download_config', u'on-error': u'node_update_failed', u'on-success': u'get_private_key', u'version': u'2.0', u'action': u'tripleo.config.download_config', u'input': {u'work_dir': u'<% $.work_dir %>/<% execution().id %>'}, u'type': u'direct'}}, u'name': u'update_nodes', u'tags': [u'tripleo-common-managed'], u'version': u'2.0', u'input': [{u'node_user': u'heat-admin'}, u'nodes', u'playbook', u'inventory_file', {u'ansible_queue_name': u'tripleo'}, {u'module_path': u'/usr/share/ansible-modules'}, {u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False', u'ANSIBLE_LOG_PATH': u'/var/log/mistral/package_update.log'}}, {u'verbosity': 1}, {u'work_dir': u'/var/lib/mistral'}, {u'skip_tags': u''}], u'description': u'Take a container and perform an update nodes by nodes'}, u'params': {u'namespace': u''}, u'input': {u'inventory_file': u'undercloud:\n hosts:\n localhost: {}\n vars:\n ansible_connection: local\n auth_url: https://10.9.65.111:13000/\n cacert: null\n os_auth_token: gAAAAABbP5CtW-INX8Drn-1A6PI3R0iNRf8I4bWd-c2OVu6rhW1O0j-EjLsfiNzA0P5BL4ZNR45kyHM4WNahCNGfFDaxa4mkNFyXTVA14W0MxwkMHxTfWGi8OciVcBnIMb0V2acp5xJ7swFM8_BKkvUGnZIepCd_91qxo9DA4eJ6V63JWqYtRSc\n overcloud_admin_password: GpW2UAqzFfEc36YzMAfj3NXZk\n overcloud_horizon_url: https://10.9.65.170:443/dashboard\n overcloud_keystone_url: https://10.9.65.170:13000/\n plan: overcloud\n project_name: admin\n undercloud_service_list: [openstack-nova-compute, openstack-heat-engine, openstack-ironic-conductor,\n openstack-swift-container, openstack-swift-object, openstack-mistral-engine]\n undercloud_swift_url: https://10.9.65.111:13808/v1/AUTH_8cab270eccb74faf9c35bca7dad0a6cb\n username: admin\novercloud-controller-0:\n hosts:\n 172.16.0.101: {}\n vars:\n ctlplane_ip: 172.16.0.101\n deploy_server_id: afb6d2a8-0937-488b-85dd-157ac38ad6bf\n enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n tenant]\n Scrolling back up, I see this: u' "+ echo \'Running command: \'\\\\\'\'/usr/bin/bootstrap_host_exec gnocchi_api /usr/bin/gnocchi-upgrade --sacks-number=128\'\\\\\'\'\'", ', u' "+ exec /usr/bin/bootstrap_host_exec gnocchi_api /usr/bin/gnocchi-upgrade --sacks-number=128", ', u' "2018-07-06 16:47:58,529 [19] WARNING oslo_config.cfg: Option \\"coordination_url\\" from group \\"storage\\" is deprecated. Use option \\"coordination_url\\" from group \\"DEFAULT\\".", ', u' "2018-07-06 16:47:58,530 [19] INFO gnocchi.service: Gnocchi version 4.2.3", ', u' "2018-07-06 16:47:59,120 [19] INFO gnocchi.cli.manage: Upgrading indexer SQLAlchemyIndexer: mysql+pymysql://gnocchi:DVmtaW8AKKchYgAmVq7K4gW8K.1.16/gnocchi?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf", ', u' "2018-07-06 16:47:59,740 [19] INFO gnocchi.cli.manage: Upgrading storage SwiftStorage: gnocchi", ', u' "2018-07-06 16:47:59,743 [19] INFO gnocchi.cli.manage: Upgrading incoming storage SwiftStorage", ', u' "2018-07-06 16:48:03,530 [19] CRITICAL root: Traceback (most recent call last):", ', u' " File \\"/usr/bin/gnocchi-upgrade\\", line 10, in <module>", ', u' " sys.exit(upgrade())", ', u' " File \\"/usr/lib/python2.7/site-packages/gnocchi/cli/manage.py\\", line 73, in upgrade", ', u' " i.upgrade(conf.sacks_number)", ', u' " File \\"/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py\\", line 71, in upgrade", ', u' " self.set_storage_settings(num_sacks)", ', u' " File \\"/usr/lib/python2.7/site-packages/gnocchi/incoming/swift.py\\", line 47, in set_storage_settings", ', u' " self.swift.put_container(self.CFG_PREFIX)", ', u' " File \\"/usr/lib/python2.7/site-packages/swiftclient/client.py\\", line 1773, in put_container", ', u' " query_string=query_string)", ', u' " File \\"/usr/lib/python2.7/site-packages/swiftclient/client.py\\", line 1691, in _retry", ', u' " service_token=self.service_token, **kwargs)", ', u' " File \\"/usr/lib/python2.7/site-packages/swiftclient/client.py\\", line 1030, in put_container", ', u' " conn.request(method, path, \'\', headers)", ', u' " File \\"/usr/lib/python2.7/site-packages/swiftclient/client.py\\", line 439, in request", ', u' " files=files, **self.requests_args)", ', u' " File \\"/usr/lib/python2.7/site-packages/swiftclient/client.py\\", line 422, in _request", ', u' " return self.request_session.request(*arg, **kwarg)", ', u' " File \\"/usr/lib/python2.7/site-packages/requests/sessions.py\\", line 518, in request", ', u' " resp = self.send(prep, **send_kwargs)", ', u' " File \\"/usr/lib/python2.7/site-packages/requests/sessions.py\\", line 639, in send", ', u' " r = adapter.send(request, **kwargs)", ', u' " File \\"/usr/lib/python2.7/site-packages/requests/adapters.py\\", line 488, in send", ', u' " raise ConnectionError(err, request=request)", ', u' "ConnectionError: (\'Connection aborted.\', BadStatusLine(\\"\'\'\\",))", ' Even further up, I see: u' "Error running [\'docker\', \'run\', \'--name\', \'gnocchi_db_sync\', \'--label\', \'config_id=tripleo_step4\', \'--label\', \'container_name=gnocchi_db_sync\', \'--label\', \'managed_by=paunch\', \'--label\', \'config_data={\\"environment\\": [\\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\\", \\"TRIPLEO_CONFIG_HASH=b156bf4f68a6fff382738d4dac4bde55\\"], \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/var/lib/kolla/config_files/gnocchi_db_sync.json:/var/lib/kolla/config_files/config.json:ro\\", \\"/var/lib/config-data/puppet-generated/gnocchi/:/var/lib/kolla/config_files/src:ro\\", \\"/var/lib/gnocchi:/var/lib/gnocchi:rw\\", \\"/var/log/containers/gnocchi:/var/log/gnocchi\\", \\"/var/log/containers/httpd/gnocchi-api:/var/log/httpd\\", \\"/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro\\"], \\"image\\": \\"172.16.0.11:8787/rhosp13/openstack-gnocchi-api:13.0-38\\", \\"detach\\": false, \\"net\\": \\"host\\", \\"privileged\\": false}\', \'--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\', \'--env=TRIPLEO_CONFIG_HASH=b156bf4f68a6fff382738d4dac4bde55\', \'--net=host\', \'--privileged=false\', \'--user=root\', \'--volume=/etc/hosts:/etc/hosts:ro\', \'--volume=/etc/localtime:/etc/localtime:ro\', \'--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\', \'--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\', \'--volume=/dev/log:/dev/log\', \'--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/var/lib/kolla/config_files/gnocchi_db_sync.json:/var/lib/kolla/config_files/config.json:ro\', \'--volume=/var/lib/config-data/puppet-generated/gnocchi/:/var/lib/kolla/config_files/src:ro\', \'--volume=/var/lib/gnocchi:/var/lib/gnocchi:rw\', \'--volume=/var/log/containers/gnocchi:/var/log/gnocchi\', \'--volume=/var/log/containers/httpd/gnocchi-api:/var/log/httpd\', \'--volume=/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro\', \'172.16.0.11:8787/rhosp13/openstack-gnocchi-api:13.0-38\']. [1]", ', It does not look like there are any Gnocchi containers on the controllers: [stack@ds-hf-ca-undercloud ansible]$ ansible controllers -m shell -b -a "hostname; docker ps | grep gnoc" 172.16.0.105 | FAILED | rc=1 >> overcloud-controller-2non-zero return code 172.16.0.101 | FAILED | rc=1 >> overcloud-controller-0non-zero return code 172.16.0.108 | FAILED | rc=1 >> overcloud-controller-1non-zero return code [stack@ds-hf-ca-undercloud ansible]$ Will attach sosreport from controllers and director. How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1457062 [details] controller-0 sosreport
Created attachment 1457063 [details] controller-1 sosreport
Created attachment 1457064 [details] controller-2 sosreport
Gnocchi containers look like they are on the undercloud: (undercloud) [stack@ds-hf-ca-undercloud ~]$ curl -X GET http://172.16.0.11:8787/v2/_catalog {"repositories":["rhosp13/openstack-aodh-api","rhosp13/openstack-aodh-evaluator","rhosp13/openstack-aodh-listener","rhosp13/openstack-aodh-notifier","rhosp13/openstack-ceilometer-central","rhosp13/openstack-ceilometer-compute","rhosp13/openstack-ceilometer-notification","rhosp13/openstack-cinder-api","rhosp13/openstack-cinder-scheduler","rhosp13/openstack-cinder-volume","rhosp13/openstack-cron","rhosp13/openstack-glance-api","rhosp13/openstack-gnocchi-api","rhosp13/openstack-gnocchi-metricd","rhosp13/openstack-gnocchi-statsd","rhosp13/openstack-haproxy","rhosp13/openstack-heat-api","rhosp13/openstack-heat-api-cfn","rhosp13/openstack-heat-engine","rhosp13/openstack-horizon","rhosp13/openstack-iscsid","rhosp13/openstack-keystone","rhosp13/openstack-mariadb","rhosp13/openstack-memcached","rhosp13/openstack-neutron-dhcp-agent","rhosp13/openstack-neutron-l3-agent","rhosp13/openstack-neutron-metadata-agent","rhosp13/openstack-neutron-openvswitch-agent","rhosp13/openstack-neutron-server","rhosp13/openstack-nova-api","rhosp13/openstack-nova-compute","rhosp13/openstack-nova-conductor","rhosp13/openstack-nova-consoleauth","rhosp13/openstack-nova-libvirt","rhosp13/openstack-nova-novncproxy","rhosp13/openstack-nova-placement-api","rhosp13/openstack-nova-scheduler","rhosp13/openstack-panko-api","rhosp13/openstack-rabbitmq","rhosp13/openstack-redis","rhosp13/openstack-swift-account","rhosp13/openstack-swift-container","rhosp13/openstack-swift-object","rhosp13/openstack-swift-proxy-server"]} (undercloud) [stack@ds-hf-ca-undercloud ~]$
Leaving here as a reference an open Gnocchi issue which shows the same ConnectionError: (\'Connection aborted.\', BadStatusLine(\\"\'\'\\",))" error: https://github.com/gnocchixyz/gnocchi/issues/509
Re-opening this as it happened again on another run through.
Darin hit this issue again today and after investigating the failed environment the root cause is that gnocchi_db_sync container runs before the swift_proxy container which obviously fails as the storage backend for gnocchi is swift which is not available at that moment. We need to start gnocchi_db_sync only after the swift_proxy container becomes available. Start time of the swift_proxy container: 19:52:02 [root@overcloud-controller-0 ~]# docker inspect swift_proxy | grep StartedAt "StartedAt": "2018-07-10T19:52:02.828526308Z", Start time of the gnocchi db sync failure: 19:51:39 2018-07-10 19:51:34,456 [19] INFO gnocchi.service: Gnocchi version 4.2.3 2018-07-10 19:51:36,896 [19] INFO gnocchi.cli.manage: Upgrading indexer SQLAlchemyIndexer: mysql+pymysql://gnocchi:DVmtaW8AKKchYgAmVq7K4gW8K.1.16/gnocchi?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf 2018-07-10 19:51:37,505 [19] INFO gnocchi.cli.manage: Upgrading storage SwiftStorage: gnocchi 2018-07-10 19:51:37,511 [19] INFO gnocchi.cli.manage: Upgrading incoming storage SwiftStorage 2018-07-10 19:51:39,998 [19] CRITICAL root: Traceback (most recent call last): File "/usr/bin/gnocchi-upgrade", line 10, in <module> sys.exit(upgrade()) File "/usr/lib/python2.7/site-packages/gnocchi/cli/manage.py", line 73, in upgrade i.upgrade(conf.sacks_number) File "/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py", line 71, in upgrade self.set_storage_settings(num_sacks) File "/usr/lib/python2.7/site-packages/gnocchi/incoming/swift.py", line 47, in set_storage_settings self.swift.put_container(self.CFG_PREFIX) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1773, in put_container query_string=query_string) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1030, in put_container conn.request(method, path, '', headers) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 439, in request files=files, **self.requests_args) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 422, in _request return self.request_session.request(*arg, **kwarg) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 488, in send raise ConnectionError(err, request=request) ConnectionError: ('Connection aborted.', BadStatusLine("''",)) Swift logs around that time: [root@overcloud-controller-0 ~]# grep -v haproxy /var/log/containers/swift/swift.log | grep 'Jul 10 19:5[1-2]' Jul 10 19:51:21 overcloud-controller-0 container-server: Starting 1 Jul 10 19:51:23 overcloud-controller-0 object-server: Started child 26 Jul 10 19:51:23 overcloud-controller-0 object-server: Started child 27 Jul 10 19:51:26 overcloud-controller-0 object-server: Starting 1 Jul 10 19:51:26 overcloud-controller-0 object-server: Begin object audit "forever" mode (ZBF) Jul 10 19:51:26 overcloud-controller-0 object-server: Object audit (ZBF). Since Tue Jul 10 19:51:26 2018: Locally: 1 passed, 0 quarantined, 0 errors, files/sec: 370.95, bytes/sec: 0.00, Total time: 0.00, Auditing time: 0.00, Rate: 0.00 Jul 10 19:51:26 overcloud-controller-0 object-server: Begin object audit "forever" mode (ALL) Jul 10 19:51:27 overcloud-controller-0 account-server: Starting 1 Jul 10 19:51:39 overcloud-controller-0 object-server: Object audit (ZBF) "forever" mode completed: 12.76s. Total quarantined: 0, Total errors: 0, Total files/sec: 50.14, Total bytes/sec: 0.00, Auditing time: 12.49, Rate: 0.98 Jul 10 19:51:41 overcloud-controller-0 account-server: Starting 1 Jul 10 19:51:46 overcloud-controller-0 container-server: Option allow_versions is deprecated. Configure the versioned_writes middleware in the proxy-server instead. This option will be ignored in a future release. Jul 10 19:51:46 overcloud-controller-0 container-server: Option allow_versions is deprecated. Configure the versioned_writes middleware in the proxy-server instead. This option will be ignored in a future release. Jul 10 19:51:46 overcloud-controller-0 container-server: Started child 20 Jul 10 19:51:46 overcloud-controller-0 container-server: Started child 21 Jul 10 19:51:46 overcloud-controller-0 object-server: Object audit (ALL). Since Tue Jul 10 19:51:26 2018: Locally: 1 passed, 0 quarantined, 0 errors, files/sec: 0.05, bytes/sec: 9998697.01, Total time: 20.48, Auditing time: 0.00, Rate: 0.00 Jul 10 19:51:47 overcloud-controller-0 container-server: Option allow_versions is deprecated. Configure the versioned_writes middleware in the proxy-server instead. This option will be ignored in a future release. Jul 10 19:51:47 overcloud-controller-0 container-server: Option allow_versions is deprecated. Configure the versioned_writes middleware in the proxy-server instead. This option will be ignored in a future release. Jul 10 19:51:47 overcloud-controller-0 container-server: Option allow_versions is deprecated. Configure the versioned_writes middleware in the proxy-server instead. This option will be ignored in a future release. Jul 10 19:51:47 overcloud-controller-0 container-server: Option allow_versions is deprecated. Configure the versioned_writes middleware in the proxy-server instead. This option will be ignored in a future release. Jul 10 19:51:48 overcloud-controller-0 object-server: Starting 1 Jul 10 19:51:48 overcloud-controller-0 object-server: Starting object replicator in daemon mode. Jul 10 19:51:48 overcloud-controller-0 object-server: Starting object replication pass. Jul 10 19:51:54 overcloud-controller-0 account-server: Starting 1 Jul 10 19:51:55 overcloud-controller-0 account-server: Beginning replication run Jul 10 19:51:55 overcloud-controller-0 account-server: ERROR reading HTTP response from {'index': 1, u'replication_port': 6002, u'weight': 100.0, u'zone': 1, u'ip': u'172.16.4.13', u'region': 1, u'id': 2, u'replication_ip': u'172.16.4.13', u'meta': u'', u'device': u'd1', u'port': 6002}: Connection refused Jul 10 19:51:55 overcloud-controller-0 account-server: ERROR reading HTTP response from {'index': 2, u'replication_port': 6002, u'weight': 100.0, u'zone': 1, u'ip': u'172.16.4.18', u'region': 1, u'id': 1, u'replication_ip': u'172.16.4.18', u'meta': u'', u'device': u'd1', u'port': 6002}: Connection refused Jul 10 19:51:55 overcloud-controller-0 account-server: Replication run OVER Jul 10 19:51:55 overcloud-controller-0 account-server: Attempted to replicate 1 dbs in 0.00957 seconds (104.49079/s) Jul 10 19:51:55 overcloud-controller-0 account-server: Removed 0 dbs Jul 10 19:51:55 overcloud-controller-0 account-server: 0 successes, 2 failures Jul 10 19:51:55 overcloud-controller-0 account-server: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:0 remote_merge:0 rsync:0 ts_repl:0 Jul 10 19:51:56 overcloud-controller-0 object-server: Starting 1 Jul 10 19:51:56 overcloud-controller-0 container-server: Starting 1 Jul 10 19:51:57 overcloud-controller-0 object-expirer: Starting 1 Jul 10 19:51:58 overcloud-controller-0 container-server: Starting 1 Jul 10 19:52:02 overcloud-controller-0 account-server: Started child 20 Jul 10 19:52:02 overcloud-controller-0 account-server: Started child 21 Jul 10 19:52:09 overcloud-controller-0 object-server: Begin object audit "forever" mode (ZBF) Jul 10 19:52:09 overcloud-controller-0 object-server: Object audit (ZBF). Since Tue Jul 10 19:52:09 2018: Locally: 1 passed, 0 quarantined, 0 errors, files/sec: 5.51, bytes/sec: 0.00, Total time: 0.18, Auditing time: 0.00, Rate: 0.00 Jul 10 19:52:10 overcloud-controller-0 container-server: Beginning replication run Jul 10 19:52:12 overcloud-controller-0 container-server: Replication run OVER Jul 10 19:52:12 overcloud-controller-0 container-server: Attempted to replicate 26 dbs in 1.84044 seconds (14.12709/s) Jul 10 19:52:12 overcloud-controller-0 container-server: Removed 0 dbs Jul 10 19:52:12 overcloud-controller-0 container-server: 52 successes, 0 failures Jul 10 19:52:12 overcloud-controller-0 container-server: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:52 remote_merge:0 rsync:0 ts_repl:0 Jul 10 19:52:13 overcloud-controller-0 proxy-server: Adding required filter listing_formats to pipeline at position 4 Jul 10 19:52:13 overcloud-controller-0 proxy-server: Adding required filter gatekeeper to pipeline at position 1 Jul 10 19:52:13 overcloud-controller-0 proxy-server: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache listing_formats ratelimit bulk tempurl formpost authtoken keystone staticweb copy container_quotas account_quotas slo dlo versioned_writes proxy-logging proxy-server". Jul 10 19:52:13 overcloud-controller-0 proxy-server: object_post_as_copy=true is deprecated; This option is now ignored Jul 10 19:52:13 overcloud-controller-0 proxy-server: Starting Keystone auth_token middleware Jul 10 19:52:13 overcloud-controller-0 proxy-server: AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. Jul 10 19:52:13 overcloud-controller-0 proxy-server: Using /var/cache/swift as cache directory for signing certificate Jul 10 19:52:13 overcloud-controller-0 proxy-server: signing_dir mode is 0755 instead of 0700 Jul 10 19:52:13 overcloud-controller-0 proxy-server: Started child 26 Jul 10 19:52:13 overcloud-controller-0 proxy-server: Started child 27 Jul 10 19:52:15 overcloud-controller-0 proxy-server: Adding required filter listing_formats to pipeline at position 4 Jul 10 19:52:15 overcloud-controller-0 proxy-server: Adding required filter gatekeeper to pipeline at position 1 Jul 10 19:52:15 overcloud-controller-0 proxy-server: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache listing_formats ratelimit bulk tempurl formpost authtoken keystone staticweb copy container_quotas account_quotas slo dlo versioned_writes proxy-logging proxy-server". Jul 10 19:52:15 overcloud-controller-0 proxy-server: Adding required filter listing_formats to pipeline at position 4 Jul 10 19:52:15 overcloud-controller-0 proxy-server: Adding required filter gatekeeper to pipeline at position 1 Jul 10 19:52:15 overcloud-controller-0 proxy-server: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache listing_formats ratelimit bulk tempurl formpost authtoken keystone staticweb copy container_quotas account_quotas slo dlo versioned_writes proxy-logging proxy-server". Jul 10 19:52:15 overcloud-controller-0 proxy-server: object_post_as_copy=true is deprecated; This option is now ignored Jul 10 19:52:15 overcloud-controller-0 proxy-server: Starting Keystone auth_token middleware Jul 10 19:52:15 overcloud-controller-0 proxy-server: object_post_as_copy=true is deprecated; This option is now ignored Jul 10 19:52:15 overcloud-controller-0 proxy-server: Starting Keystone auth_token middleware Jul 10 19:52:16 overcloud-controller-0 proxy-server: AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. Jul 10 19:52:16 overcloud-controller-0 proxy-server: AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True. Jul 10 19:52:16 overcloud-controller-0 proxy-server: Using /var/cache/swift as cache directory for signing certificate Jul 10 19:52:16 overcloud-controller-0 proxy-server: signing_dir mode is 0755 instead of 0700 Jul 10 19:52:16 overcloud-controller-0 proxy-server: Using /var/cache/swift as cache directory for signing certificate Jul 10 19:52:16 overcloud-controller-0 proxy-server: signing_dir mode is 0755 instead of 0700 Jul 10 19:52:22 overcloud-controller-0 object-server: Object audit (ZBF) "forever" mode completed: 12.95s. Total quarantined: 0, Total errors: 0, Total files/sec: 49.43, Total bytes/sec: 0.00, Auditing time: 12.32, Rate: 0.95 Jul 10 19:52:25 overcloud-controller-0 account-server: Beginning replication run Jul 10 19:52:25 overcloud-controller-0 account-server: Replication run OVER Jul 10 19:52:25 overcloud-controller-0 account-server: Attempted to replicate 1 dbs in 0.03578 seconds (27.94853/s) Jul 10 19:52:25 overcloud-controller-0 account-server: Removed 0 dbs Jul 10 19:52:25 overcloud-controller-0 account-server: 2 successes, 0 failures Jul 10 19:52:25 overcloud-controller-0 account-server: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:2 remote_merge:0 rsync:0 ts_repl:0 Jul 10 19:52:31 overcloud-controller-0 object-server: 971/971 (100.00%) partitions replicated in 43.38s (22.38/sec, 0s remaining) Jul 10 19:52:31 overcloud-controller-0 object-server: 1942 successes, 0 failures Jul 10 19:52:31 overcloud-controller-0 object-server: 640 suffixes checked - 0.00% hashed, 0.00% synced Jul 10 19:52:31 overcloud-controller-0 object-server: Partition times: max 1.2668s, min 0.0058s, med 0.0340s Jul 10 19:52:31 overcloud-controller-0 object-server: Object replication complete. (0.72 minutes) Jul 10 19:52:40 overcloud-controller-0 container-server: Beginning replication run Jul 10 19:52:41 overcloud-controller-0 container-server: Replication run OVER Jul 10 19:52:41 overcloud-controller-0 container-server: Attempted to replicate 26 dbs in 0.87985 seconds (29.55065/s) Jul 10 19:52:41 overcloud-controller-0 container-server: Removed 0 dbs Jul 10 19:52:41 overcloud-controller-0 container-server: 52 successes, 0 failures Jul 10 19:52:41 overcloud-controller-0 container-server: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:52 remote_merge:0 rsync:0 ts_repl:0 Jul 10 19:52:52 overcloud-controller-0 object-server: Begin object audit "forever" mode (ZBF) Jul 10 19:52:52 overcloud-controller-0 object-server: Object audit (ZBF). Since Tue Jul 10 19:52:52 2018: Locally: 1 passed, 0 quarantined, 0 errors, files/sec: 406.35, bytes/sec: 0.00, Total time: 0.00, Auditing time: 0.00, Rate: 0.00 Jul 10 19:52:55 overcloud-controller-0 account-server: Beginning replication run Jul 10 19:52:55 overcloud-controller-0 account-server: Replication run OVER Jul 10 19:52:55 overcloud-controller-0 account-server: Attempted to replicate 1 dbs in 0.02268 seconds (44.09931/s) Jul 10 19:52:55 overcloud-controller-0 account-server: Removed 0 dbs Jul 10 19:52:55 overcloud-controller-0 account-server: 2 successes, 0 failures Jul 10 19:52:55 overcloud-controller-0 account-server: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:2 remote_merge:0 rsync:0 ts_repl:0 [root@overcloud-controller-0 ~]#
Let me know when we get some draft text here. I'll regenerate the OSP13 release notes include it.
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible. If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-". To add draft documentation text: * Select the documentation type from the "Doc Type" drop down field. * A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2574