Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1859193

Summary: time out when remove 300 bridges and vlans
Product: Red Hat Enterprise Linux 8 Reporter: Mingyu Shi <mshi>
Component: nmstateAssignee: Gris Ge <fge>
Status: CLOSED ERRATA QA Contact: Mingyu Shi <mshi>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.3CC: ferferna, fge, jiji, jishi, network-qe, till
Target Milestone: rcKeywords: Triaged
Target Release: 8.4Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 15:17:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
gen_state.py
none
error log
none
error log in new version
none
error log
none
gen_state.py
none
gen_state.py none

Description Mingyu Shi 2020-07-21 12:16:04 UTC
Created attachment 1701883 [details]
gen_state.py

Description of problem:
Verified #bz1820009 , got timeout failure when bring them absent. 
Another problem is: when it was timeout, it didn't fallback, means some interfaces were removed and some were not.

Version-Release number of selected component (if applicable):
nmstate-0.3.3-2.el8.noarch
NetworkManager-1.26.0-2.el8.x86_64
DISTRO=RHEL-8.3.0-20200721.n.0
Linux dell-r330-15.gsslab.brq.redhat.com 4.18.0-226.el8.x86_64 #1 SMP Wed Jul 15 07:40:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

How reproducible:
100%

Steps to Reproduce:
1. ./gen_state.py 300 up > 300.yaml
2. nmstatectl set 300.yaml
3. sed 's/state: up/state: absent/' 300.yaml | nmstatectl set

Actual results:
timeout and partial interfaces were removed

Expected results:
All removed

Additional info:

Comment 1 Mingyu Shi 2020-07-21 12:17:52 UTC
Created attachment 1701884 [details]
error log

Comment 2 Gris Ge 2020-07-27 03:30:04 UTC
In my VM, creating 300 bridge over VLAN each takes 55 seconds.

After all bridge stp finished, mark them as absent takes 28 seconds.


Can you try again with nmstate-0.3.4-1.el8?

Thanks.

Comment 3 Mingyu Shi 2020-08-06 11:17:24 UTC
Gris, I tried with 
nmstate-0.3.4-9.el8.noarch
NetworkManager-1.26.0-2.el8.x86_64
DISTRO=RHEL-8.3.0-20200728.n.0
Linux dell-r330-15.gsslab.brq.redhat.com 4.18.0-228.el8.x86_64 #1 SMP Mon Jul 27 06:05:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Started at:
2020-08-06 19:04:58,198 root         WARNING  Failed to load plugin nmstate_plugin_ovsdb: No module named 'ovs'
2020-08-06 19:04:59,295 root         DEBUG    Async action: Create checkpoint started

The last time and info:
2020-08-06 19:05:41,277 root         DEBUG    Async action: Rollback to checkpoint /org/freedesktop/NetworkManager/Checkpoint/14 finished
Traceback (most recent call last):
  File "/usr/bin/nmstatectl", line 11, in <module>
    load_entry_point('nmstate==0.3.4', 'console_scripts', 'nmstatectl')()
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 67, in main
    return args.func(args)
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 267, in apply
    args.save_to_disk,
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 289, in apply_state
    save_to_disk=save_to_disk,
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 71, in apply
    _apply_ifaces_state(plugins, net_state, verify_change, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 104, in _apply_ifaces_state
    plugin.apply_changes(net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/plugin.py", line 178, in apply_changes
    nm_applier.apply_changes(self.context, net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 154, in apply_changes
    _set_ifaces_admin_state(context, ifaces_desired_state, con_profiles)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 352, in _set_ifaces_admin_state
    device.deactivate(context, dev)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/device.py", line 47, in deactivate
    act_con.deactivate()
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/active_connection.py", line 77, in deactivate
    self._ctx.register_async(action)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/context.py", line 141, in register_async
    self.wait_all_finish()
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/context.py", line 216, in wait_all_finish
    raise tmp_error
libnmstate.error.NmstateTimeoutError: Action {'Deactivate profile: dummy0.279'} timeout

Comment 4 Mingyu Shi 2020-08-06 13:14:48 UTC
Created attachment 1710652 [details]
error log in new version

Here's the whole nmstate log

Comment 5 Gris Ge 2020-08-07 03:28:55 UTC
Thanks. I have reproduced this problem by using dummy interface instead of veth.

Investigating

Comment 6 Gris Ge 2020-08-07 08:09:49 UTC
This only impact dummy interface and waiting NetworkManager team's investigation.

Comment 7 Gris Ge 2020-09-01 15:03:39 UTC
Confirmed as bug of nmstate and patch sent to upstream.

Comment 11 Mingyu Shi 2020-10-30 02:11:19 UTC
Created attachment 1725200 [details]
error log

Using nmstate-0.4.1-1.el8.noarch, still got a timeout on 
Linux hp-dl388g8-04.rhts.eng.pek2.redhat.com 4.18.0-240.8.el8.x86_64 #1 SMP Sun Oct 25 16:56:23 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
but the error message is different

start 09:57:13
end 09:59:28

Comment 13 Fernando F. Mancera 2021-01-12 11:26:00 UTC
Hi Mingyu, could you try this out using nmstate-1.0.0? I couldn't reproduce it using it. Thanks!

Comment 14 Mingyu Shi 2021-01-13 02:46:15 UTC
It works well with
nmstate-1.0.1-0.1.el8.noarch
nispor-1.0.1-2.el8.x86_64
NetworkManager-1.30.0-0.5.el8.x86_64

Comment 15 Mingyu Shi 2021-01-13 05:57:02 UTC
Created attachment 1746904 [details]
gen_state.py

Recreate gen_state.py as the older one contained Windows style EOL

Comment 17 Mingyu Shi 2021-01-20 07:37:39 UTC
Created attachment 1748961 [details]
gen_state.py

Try again

Comment 18 Mingyu Shi 2021-02-01 11:09:57 UTC
Verified with versions:
nmstate-1.0.2-0.1.el8.noarch
nispor-1.0.1-2.el8.x86_64
NetworkManager-1.30.0-0.8.el8.x86_64

Comment 20 errata-xmlrpc 2021-05-18 15:17:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1748