Bug 1781363

Summary:	[OSP-16] Bump pyroute2 To 0.5.6 to fix concurrency issues
Product:	Red Hat OpenStack	Reporter:	Bernard Cafarelli <bcafarel>
Component:	python-pyroute2	Assignee:	Bernard Cafarelli <bcafarel>
Status:	CLOSED ERRATA	QA Contact:	Eran Kuris <ekuris>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	16.0 (Train)	CC:	amuller, atragler, ccamposr, chrisw, scohen, shrjoshi
Target Milestone:	rc	Keywords:	AutomationBlocker, Triaged
Target Release:	16.0 (Train on RHEL 8.1)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-pyroute2-0.5.6-1.el8ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-06 14:43:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Bernard Cafarelli 2019-12-09 21:02:20 UTC

Description of problem:
While functional tests pass now from time to time, most times the run will end with 1-4 failing tests in varying tests (and various errors)

Version-Release number of selected component (if applicable):
last osp 16 neutron sources

How reproducible:
90%

Sample failed tests recently observed:
neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_router_rule_and_route_table_cleared_when_fip_removed
neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_update_gateway_port_with_no_gw_port_in_namespace
neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_dvr_update_gateway_port_with_no_gw_port_in_namespace
neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_ha_dvr_local_router_fip_qos
neutron.tests.functional.agent.l3.extensions.qos.test_fip_qos_extension.TestL3AgentFipQosExtensionDVR.test_snat_bound_floating_ip
neutron.tests.functional.agent.l3.extensions.test_conntrack_helper_extension.TestL3AgentConntrackHelperExtension.test_ha_router_conntrack_helper
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_connection_from_same_address_scope
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_fip_and_router_namespace_rules_with_address_scopes_match
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_fip_agent_mismatch
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_fips_for_multiple_ext_networks
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_fips_stale_gw_port
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_rule_and_route_table_cleared_when_fip_removed
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_router_with_centralized_fip_calls_keepalived_cidr
neutron.tests.functional.agent.l3.extensions.test_gateway_ip_qos_extension.TestRouterGatewayIPQosAgentExtensionDVR.test_dvr_update_gateway_port_no_fip_fg_port_recovers_itself_with_fpr
neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_edge_router
neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_router_gateway_update_to_none
neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_router_snat_namespace_with_interface_remove
neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_update_gateway_port_with_no_gw_port_in_namespace
neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_prevent_snat_rule_exist_on_restarted_agent
neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_connection_from_same_address_scope
neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_router_rule_and_route_table_cleared_when_fip_removed
neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_router_snat_namespace_with_interface_remove
neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_snat_namespace_has_ip_nonlocal_bind_disabled
neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_update_gateway_port_with_no_gw_port_in_namespace
neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_update_gateway_port_with_no_gw_port_in_namespace
neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_snat_bound_floating_ip
neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_conntrack_disassociate_fip_ha_router
neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_connection_from_diff_address_scope
neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_periodic_sync_routers_task_routers_deleted_while_agent_down
neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_periodic_sync_routers_task_routers_deleted_while_agent_down
neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_periodic_sync_routers_task_routers_deleted_while_agent_down
neutron.tests.functional.agent.l3.test_metadata_proxy.UnprivilegedUserGroupMetadataL3AgentTestCase.test_access_to_metadata_proxy

Sample failures:
pyroute2.netlink.exceptions.NetlinkError: (9, 'Bad file descriptor')
neutron_lib.exceptions.ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-9f64283d-28ed-4dd9-a6ae-e19366552f4b": No such file or directory
OSError: [Errno 9] Bad file descriptor

While failing tests are quite different, most seem to have the same root cause (just initial supposition, this may have multiple reasons

Comment 2 Bernard Cafarelli 2019-12-12 15:48:11 UTC

Linking original launchpad bug about it, where functional tests (and also some tempest ones) were failing 100%. This was caused by a bug in pyroute2 < 0.5.5
Currently, OSP 16 repo has python3-pyroute2-0.5.3-4. We should match train upper-requirements and bump pyroute2 to 0.5.6.

Similarly, if RDO Train does not use this version, it should be bumped too

Comment 5 Bernard Cafarelli 2019-12-12 17:17:42 UTC

RDO already has 0.5.6 in train, nothing to update there

Comment 14 errata-xmlrpc 2020-02-06 14:43:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0283

Comment 15 Red Hat Bugzilla 2023-09-14 05:48:25 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days