Description of problem: Some tempest tests failed executing their teardown because of the following error: tempest.lib.exceptions.BadRequest: Bad request Details: {'code': 400, 'message': 'sequence item 0: expected str instance, bytes found', 'title': 'Bad Request'} Link to the tempest logs: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve-gate-ovn/90//artifact/tempest-results/tempest-results-neutron_single_threaded.1.html keystone logs: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve-gate-ovn/90/controller-0/var/log/containers/keystone/keystone.log.gz Occurrence 1: 2022-02-11 19:46:30.358 189 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-bf4a399e-d3ad-471c-bdee-3bac0a34ce95 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] RBAC: auth_context: {'token': <TokenModel (audit_id=2GrPevCfSbuaByNCjruekQ, audit_chain_id=['2GrPevCfSbuaByNCjruekQ']) at 0x7f7bf9707f98>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c0ae35fa7fb4cd88a34c7b6f51c1f43', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cfe8694106044c3dab72acd44d4df48e', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py:479 2022-02-11 19:46:30.358 189 DEBUG keystone.server.flask.request_processing.req_logging [req-bf4a399e-d3ad-471c-bdee-3bac0a34ce95 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] REQUEST_METHOD: `DELETE` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:27 2022-02-11 19:46:30.359 189 DEBUG keystone.server.flask.request_processing.req_logging [req-bf4a399e-d3ad-471c-bdee-3bac0a34ce95 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] SCRIPT_NAME: `` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:28 2022-02-11 19:46:30.359 189 DEBUG keystone.server.flask.request_processing.req_logging [req-bf4a399e-d3ad-471c-bdee-3bac0a34ce95 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] PATH_INFO: `/v3/users/3fed504c9abd4e1685e29d61435fec4b` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:29 2022-02-11 19:46:30.360 189 DEBUG keystone.common.rbac_enforcer.enforcer [req-bf4a399e-d3ad-471c-bdee-3bac0a34ce95 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] RBAC: Authorizing `identity:delete_user(user_id=3fed504c9abd4e1685e29d61435fec4b)` enforce_call /usr/lib/python3.6/site-packages/keystone/common/rbac_enforcer/enforcer.py:442 2022-02-11 19:46:30.378 189 DEBUG keystone.common.rbac_enforcer.enforcer [req-bf4a399e-d3ad-471c-bdee-3bac0a34ce95 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] RBAC: Authorization granted enforce_call /usr/lib/python3.6/site-packages/keystone/common/rbac_enforcer/enforcer.py:449 2022-02-11 19:46:30.427 189 WARNING keystone.server.flask.application [req-bf4a399e-d3ad-471c-bdee-3bac0a34ce95 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] sequence item 0: expected str instance, bytes found Ocurrence 2: 2022-02-11 19:56:00.745 185 DEBUG keystone.server.flask.request_processing.middleware.auth_context [req-e1147925-db8a-44cc-b949-e1e705ef6339 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] RBAC: auth_context: {'token': <TokenModel (audit_id=zv_FdcRZTFSeyEYQ6DPPiA, audit_chain_id=['zv_FdcRZTFSeyEYQ6DPPiA']) at 0x7f7bf8517240>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7c0ae35fa7fb4cd88a34c7b6f51c1f43', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cfe8694106044c3dab72acd44d4df48e', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} fill_context /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/middleware/auth_context.py:479 2022-02-11 19:56:00.746 185 DEBUG keystone.server.flask.request_processing.req_logging [req-e1147925-db8a-44cc-b949-e1e705ef6339 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] REQUEST_METHOD: `DELETE` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:27 2022-02-11 19:56:00.746 185 DEBUG keystone.server.flask.request_processing.req_logging [req-e1147925-db8a-44cc-b949-e1e705ef6339 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] SCRIPT_NAME: `` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:28 2022-02-11 19:56:00.746 185 DEBUG keystone.server.flask.request_processing.req_logging [req-e1147925-db8a-44cc-b949-e1e705ef6339 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] PATH_INFO: `/v3/users/8e97e83a70414a28b2c1417ebedbe726` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:29 2022-02-11 19:56:00.747 185 DEBUG keystone.common.rbac_enforcer.enforcer [req-e1147925-db8a-44cc-b949-e1e705ef6339 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] RBAC: Authorizing `identity:delete_user(user_id=8e97e83a70414a28b2c1417ebedbe726)` enforce_call /usr/lib/python3.6/site-packages/keystone/common/rbac_enforcer/enforcer.py:442 2022-02-11 19:56:00.770 185 DEBUG keystone.common.rbac_enforcer.enforcer [req-e1147925-db8a-44cc-b949-e1e705ef6339 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] RBAC: Authorization granted enforce_call /usr/lib/python3.6/site-packages/keystone/common/rbac_enforcer/enforcer.py:449 2022-02-11 19:56:00.822 185 WARNING keystone.server.flask.application [req-e1147925-db8a-44cc-b949-e1e705ef6339 7c0ae35fa7fb4cd88a34c7b6f51c1f43 cfe8694106044c3dab72acd44d4df48e - default default] sequence item 0: expected str instance, bytes found Version-Release number of selected component (if applicable): RHOS-16.1-RHEL-8-20220210.n.1 How reproducible: not always, but the following job reproduces it often: DFG-network-networking-ovn-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve-gate-ovn Steps to Reproduce: 1. run the job mentioned above 2. 3.
I think we didn't get a full traceback in the server logs, because a bug in keystone. I submitted a fix upstream. If you can, please apply it in the job run, so that we can get some more output.
python-memcached 1.59-2 is in the current compose which has passed phase2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8795