Description of problem: Controller Replacement regression jobs are successful until test Tempest stage. Between 30 and 40 Tempest tests fail. Example of failed Tempest tests are: Run Tempest Tests / tempest.api.object_storage.test_container_services_negative.ContainerNegativeTest.test_delete_non_empty_container[id-42da116e-1e8c-4c96-9e06-2f13884ed2b1,negative] Run Tempest Tests / tempest.api.network.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_add_remove_network_from_dhcp_agent[id-a0856713-6549-470c-a656-e97c8df9a14d] Run Tempest Tests / tempest.api.network.admin.test_dhcp_agent_scheduler.DHCPAgentSchedulersTestJSON.test_list_networks_hosted_by_one_dhcp[id-30c48f98-e45d-4ffb-841c-b8aad57c7587] Run Tempest Tests / .tearDownClass (tempest.api.object_storage.test_account_quotas.AccountQuotasTest) Run Tempest Tests / tempest.api.object_storage.test_container_quotas.ContainerQuotasTest.test_upload_too_many_objects[id-3a387039-697a-44fc-a9c0-935de31f426b,smoke] Run Tempest Tests / tempest.api.object_storage.test_container_quotas.ContainerQuotasTest.test_upload_valid_object[id-9a0fb034-86af-4df0-86fa-f8bd7db21ae0,smoke] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_end_marker[id-55b4fa5c-e12e-4ca9-8fcf-a79afe118522] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_format_json[id-196f5034-6ab0-4032-9da9-a937bbb9fba9] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_format_xml[id-655a53ca-4d15-408c-a377-f4c6dbd0a1fa] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_limit[id-297ec38b-2b61-4ff4-bcd1-7fa055e97b61] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_marker[id-c31ddc63-2a58-4f6b-b25c-94d2937e6867] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_no_object[id-4646ac2d-9bfb-4c7d-a3c5-0f527402b3df] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_path[id-58ca6cc9-6af0-408d-aaec-2a6a7b2f0df9] Run Tempest Tests / tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container[id-a151e158-dcbf-4a1f-a1e7-46cd65895a6f] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_prefix[id-77e742c7-caf2-4ec9-8aa4-f7d509a3344c] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_metadata[id-96e68f0e-19ec-4aa2-86f3-adc6a45e14dd,smoke] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_list_no_container_metadata[id-a2faf936-6b13-4f8d-92a2-c2278355821e] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_update_container_metadata_with_create_and_delete_metadata[id-cf19bc0b-7e16-4a5a-aaed-cb0c2fe8deef] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_update_container_metadata_with_create_metadata[id-2ae5f295-4bf1-4e04-bfad-21e54b62cec5] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_update_container_metadata_with_create_metadata_key[id-31f40a5f-6a52-4314-8794-cd89baed3040] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_update_container_metadata_with_delete_metadata[id-3a5ce7d4-6e4b-47d0-9d87-7cd42c325094] Run Tempest Tests / tempest.api.object_storage.test_container_services.ContainerTest.test_update_container_metadata_with_delete_metadata_key[id-a2e36378-6f1f-43f4-840a-ffd9cfd61914] Run Tempest Tests / tempest.api.object_storage.test_container_staticweb.StaticWebTest.test_web_listing_css[id-bc37ec94-43c8-4990-842e-0e5e02fc8926] Run Tempest Tests / .tearDownClass (tempest.api.object_storage.test_container_staticweb.StaticWebTest) Run Tempest Tests / tempest.scenario.test_object_storage_basic_ops.TestObjectStorageBasicOps.test_swift_basic_ops[id-b920faf1-7b8a-4657-b9fe-9c4512bfb381,object_storage] Run Tempest Tests / tempest.api.object_storage.test_object_services.ObjectTest.test_get_object_with_x_object_manifest[id-11b4515b-7ba7-4ca8-8838-357ded86fc10] Run Tempest Tests / tempest.api.object_storage.test_object_slo.ObjectSloTest.test_delete_large_object[id-87b6dfa1-abe9-404d-8bf0-6c3751e6aa77] Run Tempest Tests / tempest.api.object_storage.test_object_services.ObjectTest.test_object_upload_in_segments[id-e3e6a64a-9f50-4955-b987-6ce6767c97fb] Run Tempest Tests / tempest.api.object_storage.test_object_services.ObjectTest.test_update_object_metadata_with_create_and_remove_metadata[id-f726174b-2ded-4708-bff7-729d12ce1f84] Run Tempest Tests / tempest.api.object_storage.test_object_slo.ObjectSloTest.test_retrieve_large_object[id-49bc49bc-dd1b-4c0f-904e-d9f10b830ee8] Run Tempest Tests / tempest.api.object_storage.test_object_slo.ObjectSloTest.test_upload_manifest[id-2c3f24a6-36e8-4711-9aa2-800ee1fc7b5b] Run Tempest Tests / .tearDownClass (tempest.api.object_storage.test_object_services.ObjectTest) Run Tempest Tests / tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest.test_put_object_using_temp_url[id-9b08dade-3571-4152-8a4f-a4f2a873a735] Run Tempest Tests / tempest.api.object_storage.test_account_bulk.BulkTest.test_extract_archive[id-a407de51-1983-47cc-9f14-47c2b059413c] Run Tempest Tests / .tearDownClass (tempest.api.object_storage.test_object_temp_url_negative.ObjectTempUrlNegativeTest) Run Tempest Tests / tempest.api.object_storage.test_container_acl.ObjectTestACLs.test_read_object_with_rights[id-a3270f3f-7640-4944-8448-c7ea783ea5b6] Run Tempest Tests / tempest.api.object_storage.test_container_acl.ObjectTestACLs.test_write_object_with_rights[id-aa58bfa5-40d9-4bc3-82b4-d07f4a9e392a] Run Tempest Tests / tempest.api.object_storage.test_account_services.AccountTest.test_list_containers[id-3499406a-ae53-4f8c-b43a-133d4dc6fe3f,smoke] Run Tempest Tests / tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_limit[id-5cfa4ab2-4373-48dd-a41f-a532b12b08b2] Run Tempest Tests / tempest.api.object_storage.test_account_services.AccountTest.test_list_containers_with_marker_and_end_marker[id-ac8502c2-d4e4-4f68-85a6-40befea2ef5e] Version-Release number of selected component (if applicable): RHOS_TRUNK-16.0-RHEL-8-20200103.n.1 How reproducible: Every time the controller replacement Jenkins regression jobs are executed. Steps to Reproduce: 1. Execute DFG-df-controller_replacement-16-virthost-3cont_3comp-yes_UC_SSL-yes_OC_SSL-lvm-ipv4-geneve-replace_controller-corrupt_disk-RHELOSP-38494 job in Jenkins 2. 3. Actual results: 30 to 40 Tempest tests fail Expected results: Tempest tests complete successfully Additional info:
I found the regression in Train/RHOSP16, and opened an upstream bug [1] and proposed a patch to fix it [2]. [1] https://bugs.launchpad.net/tripleo/+bug/1892674 [2] https://review.opendev.org/#/c/747621/ This only applies to Stein and Train. I'm not sure if this is the same reason Takashi found on RHOSP13, but I will look into that next.
After debugging this further, it shows that this is not a regression, and also affects OSP13 as Takashi noticed. I updated the Launchpad bug entry and the patch on Gerrit, this needs to be applied to our downstream releases as well.
Patch merged on master, proposed backports: https://review.opendev.org/#/c/749883/ Train https://review.opendev.org/#/c/749884/ Ussuri https://review.opendev.org/#/c/749885/ Stein https://review.opendev.org/#/c/749886/ Rocky https://review.opendev.org/#/c/749887/ Queens
Yes, controller job is passing with current build.
All the storage tempest tests that originally failed during the controller replacement job now pass. RHOS-16.1-RHEL-8-20201021.n.0 was used.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:5413