Bug 2218060

Summary: [CI] Linux bridge not been deleted after its connection been deactivated
Product: Red Hat Enterprise Linux 9 Reporter: Gris Ge <fge>
Component: NetworkManagerAssignee: NetworkManager Development Team <nm-team>
Status: CLOSED MIGRATED QA Contact: Desktop QE <desktop-qa-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 9.2CC: bgalvani, lrintel, rkhan, sfaye, sukulkar, till
Target Milestone: rcKeywords: MigratedToJIRA, Regression, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-17 13:09:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
NetworkManager trace log none

Description Gris Ge 2023-06-28 03:14:36 UTC 
Created attachment 1972907 [details]
NetworkManager trace log

Description of problem:

The nmstate CI found failure on `test_create_down_linux_bridge` test case on latest NM main branch.

https://github.com/nmstate/nmstate/actions/runs/5389866750/jobs/9784645584#step:7:1284

Also confirmed this in local VM using NetworkManager-1.43.10-32444.copr.42689d1c90.el9.x86_64 after 28 retry on this single test case.

Version-Release number of selected component (if applicable):
NetworkManager-1.43.10-32444.copr.42689d1c90.el9.x86_64

How reproducible:
1%

Steps to Reproduce:
0. Install NetworkManager-1.43.10-32444.copr.42689d1c90.el9.x86_64
1. sudo dnf install nmstate python3-pytest python3-libnmstate git -y
2. git clone https://github.com/nmstate/nmstate.git
3. sudo ip netns add tmp
sudo ip link add eth2 type veth peer name eth2peer
sudo ip link add eth1 type veth peer name eth1peer
sudo ip link set eth1 up
sudo ip link set eth2 up
sudo ip link set eth1peer up
sudo ip link set eth2peer up
sudo ip link set eth1peer netns tmp
sudo ip link set eth2peer netns tmp
sudo ip netns exec tmp ip link set eth1peer up
sudo ip netns exec tmp ip link set eth2peer up

sudo nmcli device set eth1 managed yes
sudo nmcli device set eth2 managed yes

4. cd nmstate/tests/integration
5. for x in `seq 1 100`; do pytest -vvv -k test_create_down_linux_bridge||break; done

Actual results:

Test failed with 

libnmstate.error.NmstateVerificationError: Absent/Down interface brtest0/linux-bridge still found

Expected results:

No test failure

Additional info:

This random failure cannot be reproduced on NetworkManager-1.42.2-3.el9_2.x86_64 with 100 retry.

Tried to mimic the test case using `nmstatectl`, no luck with 1000 retry, my guess is bash script invoke new process to apply the state is not fast enough to trigger the problem than single process python library.