Description of problem: While functional tests pass now from time to time, most times the run will end with 1-4 failing tests in varying tests (and various errors) Version-Release number of selected component (if applicable): last osp 16 neutron sources How reproducible: 90% Sample failed tests recently observed: neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_router_rule_and_route_table_cleared_when_fip_removed neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_update_gateway_port_with_no_gw_port_in_namespace neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_update_gateway_port_with_no_gw_port_in_namespace neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_ha_dvr_local_router_fip_qos neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_snat_bound_floating_ip neutron.tests.functional.agent.l3.extensions.test_conntrack_helper_extension.TestL3AgentConntrackHelperExtension.test_ha_router_conntrack_helper neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_connection_from_same_address_scope neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_fip_and_router_namespace_rules_with_address_scopes_match neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_fip_agent_mismatch neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_fips_for_multiple_ext_networks neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_fips_stale_gw_port neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_rule_and_route_table_cleared_when_fip_removed neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_with_centralized_fip_calls_keepalived_cidr neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_update_gateway_port_no_fip_fg_port_recovers_itself_with_fpr neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_edge_router neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_router_gateway_update_to_none neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_router_snat_namespace_with_interface_remove neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_update_gateway_port_with_no_gw_port_in_namespace neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_prevent_snat_rule_exist_on_restarted_agent neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_connection_from_same_address_scope neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_router_rule_and_route_table_cleared_when_fip_removed neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_router_snat_namespace_with_interface_remove neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_snat_namespace_has_ip_nonlocal_bind_disabled neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_update_gateway_port_with_no_gw_port_in_namespace neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_update_gateway_port_with_no_gw_port_in_namespace neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_snat_bound_floating_ip neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_conntrack_disassociate_fip_ha_router neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_connection_from_diff_address_scope neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_periodic_sync_routers_task_routers_deleted_while_agent_down neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_periodic_sync_routers_task_routers_deleted_while_agent_down neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_periodic_sync_routers_task_routers_deleted_while_agent_down neutron.tests.functional.agent.l3.test_metadata_proxy.UnprivilegedUserGroupMetadataL3AgentTestCase.test_access_to_metadata_proxy Sample failures: pyroute2.netlink.exceptions.NetlinkError: (9, 'Bad file descriptor') neutron_lib.exceptions.ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-9f64283d-28ed-4dd9-a6ae-e19366552f4b": No such file or directory OSError: [Errno 9] Bad file descriptor While failing tests are quite different, most seem to have the same root cause (just initial supposition, this may have multiple reasons
Linking original launchpad bug about it, where functional tests (and also some tempest ones) were failing 100%. This was caused by a bug in pyroute2 < 0.5.5 Currently, OSP 16 repo has python3-pyroute2-0.5.3-4. We should match train upper-requirements and bump pyroute2 to 0.5.6. Similarly, if RDO Train does not use this version, it should be bumped too
RDO already has 0.5.6 in train, nothing to update there
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0283
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days