2215022 – [NMCI] crash in ovs_mtu test

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2215022 - [NMCI] crash in ovs_mtu test

Summary: [NMCI] crash in ovs_mtu test

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	NetworkManager
Sub Component:
Version:	9.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Gris Ge
QA Contact:	Vladimir Benes
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-06-14 13:34 UTC by Vladimir Benes
Modified:	2023-11-07 10:13 UTC (History)
CC List:	8 users (show)
Fixed In Version:	NetworkManager-1.43.11-1.el9
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-11-07 08:38:26 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:
Flags:	pm-rhel: mirror+

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	NMT-617	None	None	None	2023-06-14 13:35:44 UTC
Red Hat Issue Tracker	RHELPLAN-159840	None	None	None	2023-06-14 13:35:48 UTC
Red Hat Product Errata	RHBA-2023:6585	None	None	None	2023-11-07 08:38:36 UTC
freedesktop.org Gitlab	NetworkManager NetworkManager merge_requests 1691	None	merged	device: do not set MAC address on iface with index <=0	2023-07-12 15:27:49 UTC

Description Vladimir Benes 2023-06-14 13:34:41 UTC

Description of problem:
when running ovs_mtu test, there is a crash here and there.


Version-Release number of selected component (if applicable):
1.43.9-1.el9

How reproducible:
not sure

Steps to Reproduce:
1. run ovs_mtu NMCI test

Actual results:
crash

Expected results:
no crash

Additional info:
           PID: 533335 (NetworkManager)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 5 (TRAP)
     Timestamp: Tue 2023-06-13 12:26:41 EDT (13s ago)
  Command Line: /usr/sbin/NetworkManager --no-daemon
    Executable: /usr/sbin/NetworkManager
 Control Group: /system.slice/NetworkManager.service
          Unit: NetworkManager.service
         Slice: system.slice
       Boot ID: 575311ff1a6640cd99170140abfccd13
    Machine ID: 22a6a62f096f4281b5344430fafb2790
      Hostname: beaker-networkmanager-gitlab-trigger-test-upstream-4669
       Storage: /var/lib/systemd/coredump/core.NetworkManager.0.575311ff1a6640cd99170140abfccd13.533335.1686673601000000.zst (present)
  Size on Disk: 864.5K
       Message: Process 533335 (NetworkManager) of user 0 dumped core.
                
                Stack trace of thread 533335:
                #0  0x00007f63483725a7 g_logv (libglib-2.0.so.0 + 0x5a5a7)
                #1  0x00007f6348372863 g_log (libglib-2.0.so.0 + 0x5a863)
                #2  0x000055b37658cbdc _nm_g_return_if_fail_warning (NetworkManager + 0x197bdc)
                #3  0x000055b3766562be _hw_addr_set (NetworkManager + 0x2612be)
                #4  0x000055b376658483 _set_state_full (NetworkManager + 0x263483)
                #5  0x000055b376661aa3 nm_device_state_changed (NetworkManager + 0x26caa3)
                #6  0x000055b3766671a8 nm_device_set_unmanaged_by_flags (NetworkManager + 0x2721a8)
                #7  0x000055b376445c47 nm_manager_stop (NetworkManager + 0x50c47)
                #8  0x00007f634803feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #9  0x00007f634803ff60 __libc_start_main_impl (libc.so.6 + 0x3ff60)
                #10 0x000055b376446ae5 _start (NetworkManager + 0x51ae5)
                
                Stack trace of thread 533336:
                #0  0x00007f634814296f __GI___poll (libc.so.6 + 0x14296f)
                #1  0x00007f63483c203c g_main_context_poll (libglib-2.0.so.0 + 0xaa03c)
                #2  0x00007f634836a5f3 g_main_context_iteration (libglib-2.0.so.0 + 0x525f3)
                #3  0x00007f634836a641 glib_worker_main (libglib-2.0.so.0 + 0x52641)
                #4  0x00007f634839b582 g_thread_proxy (libglib-2.0.so.0 + 0x83582)
                #5  0x00007f634809f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007f634803f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 533337:
                #0  0x00007f634814296f __GI___poll (libc.so.6 + 0x14296f)
                #1  0x00007f63483c203c g_main_context_poll (libglib-2.0.so.0 + 0xaa03c)
                #2  0x00007f634836c483 g_main_loop_run (libglib-2.0.so.0 + 0x54483)
                #3  0x00007f63485bee1a gdbus_shared_thread_func (libgio-2.0.so.0 + 0x110e1a)
                #4  0x00007f634839b582 g_thread_proxy (libglib-2.0.so.0 + 0x83582)
                #5  0x00007f634809f832 start_thread (libc.so.6 + 0x9f832)
                #6  0x00007f634803f450 __clone3 (libc.so.6 + 0x3f450)
                
                Stack trace of thread 533338:
                #0  0x00007f634803ee5d syscall (libc.so.6 + 0x3ee5d)
                #1  0x00007f63483bbb6c g_cond_wait_until (libglib-2.0.so.0 + 0xa3b6c)
                #2  0x00007f634833d071 g_async_queue_pop_intern_unlocked (libglib-2.0.so.0 + 0x25071)
                #3  0x00007f634833d1f6 g_async_queue_timeout_pop (libglib-2.0.so.0 + 0x251f6)
                #4  0x00007f634839e519 g_thread_pool_wait_for_new_pool (libglib-2.0.so.0 + 0x86519)
                #5  0x00007f634839b582 g_thread_proxy (libglib-2.0.so.0 + 0x83582)
                #6  0x00007f634809f832 start_thread (libc.so.6 + 0x9f832)
                #7  0x00007f634803f450 __clone3 (libc.so.6 + 0x3f450)
                ELF object binary architecture: AMD x86-64

GNU gdb (GDB) Red Hat Enterprise Linux 10.2-11.el9
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/NetworkManager...
Reading symbols from /usr/lib/debug/usr/sbin/NetworkManager-1.43.9-1.el9.x86_64.debug...
[New LWP 533335]
[New LWP 533336]
[New LWP 533337]
[New LWP 533338]
Missing separate debuginfo for /usr/lib64/NetworkManager/1.43.9-1.el9/libnm-device-plugin-ovs.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/fc/78a05e09f3b1aa5e825435bcc718dd1bae7871.debug
Missing separate debuginfo for /usr/lib64/NetworkManager/1.43.9-1.el9/libnm-device-plugin-wifi.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/39/fcae642d5b3b2bfd5b89e71f2f73851c917b3d.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/NetworkManager --no-daemon'.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
#0  g_logv (log_domain=0x55b37668d568 "nm", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=<optimized out>) at ../glib/gmessages.c:1413
1413		  g_private_set (&g_log_depth, GUINT_TO_POINTER (depth));
[Current thread is 1 (Thread 0x7f634709d500 (LWP 533335))]
(gdb) #0  g_logv (log_domain=0x55b37668d568 "nm", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=<optimized out>) at ../glib/gmessages.c:1413
#1  0x00007f6348372863 in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>) at ../glib/gmessages.c:1451
#2  0x000055b37658cbdc in _nm_g_return_if_fail_warning (log_domain=<optimized out>, line=<optimized out>, file=<optimized out>) at ./src/libnm-glib-aux/nm-gassert-patch.h:25
#3  nm_platform_link_set_address (self=<optimized out>, ifindex=0, address=0x7ffd54ba7270, length=6) at src/libnm-platform/nm-platform.c:1862
#4  0x000055b3766562be in _hw_addr_set (self=0x55b376e066b0, addr=0x55b376e20800 "7A:F1:21:AD:A1:3E", operation=0x55b37669aa8a "reset", detail=0x55b3766ba89f "unmanage") at src/core/devices/nm-device.c:16831
#5  0x000055b376658483 in _set_state_full (self=0x55b376e066b0, state=<optimized out>, reason=<optimized out>, quitting=<optimized out>) at src/core/devices/nm-device.c:16140
#6  0x000055b376661aa3 in nm_device_state_changed (reason=NM_DEVICE_STATE_REASON_REMOVED, state=NM_DEVICE_STATE_UNMANAGED, self=0x55b376e066b0) at src/core/devices/nm-device.c:16400
#7  _set_unmanaged_flags (self=0x55b376e066b0, flags=<optimized out>, set_op=<optimized out>, allow_state_transition=<optimized out>, now=<optimized out>, reason=<optimized out>) at src/core/devices/nm-device.c:14778
#8  0x000055b3766671a8 in nm_device_set_unmanaged_by_flags (reason=<optimized out>, set_op=NM_UNMAN_FLAG_OP_SET_UNMANAGED, flags=NM_UNMANAGED_QUITTING, self=<optimized out>) at src/core/devices/nm-device.c:14817
#9  nm_device_set_unmanaged_by_quitting (self=<optimized out>) at src/core/devices/nm-device.c:14950
#10 remove_device (self=0x55b376c8e000, device=<optimized out>, quitting=1) at src/core/nm-manager.c:2286
#11 0x000055b376445c47 in nm_manager_stop (self=0x55b376c8e000) at src/core/nm-manager.c:7905
#12 main (argc=<optimized out>, argv=<optimized out>) at src/core/main.c:530
(gdb) quit

Comment 2 Vladimir Benes 2023-06-23 15:57:07 UTC

disabling the test for rhel9
https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/commit/3a7df9dff0cfde3156409585caf499f0f726e208

Comment 4 Gris Ge 2023-07-12 06:43:27 UTC

Patch sent to upstream https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1691

Comment 7 Vladimir Benes 2023-07-17 20:14:13 UTC

ovs_mtu test running happily under rhel9.3 again.

Comment 9 errata-xmlrpc 2023-11-07 08:38:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6585

Note You need to log in before you can comment on or make changes to this bug.