2027674 – nm-cloud-setup[669]: GError set over the top of a previous GError or uninitialized memory

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2027674 - nm-cloud-setup[669]: GError set over the top of a previous GError or uninitialized memory

Summary: nm-cloud-setup[669]: GError set over the top of a previous GError or uninitia...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	NetworkManager
Sub Component:
Version:	9.0
Hardware:	All
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Wen Liang
QA Contact:	Desktop QE
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-11-30 12:25 UTC by Frank Liang
Modified:	2022-11-15 12:07 UTC (History)
CC List:	13 users (show)
Fixed In Version:	NetworkManager-1.37.2-1.el9
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-11-15 10:49:31 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:
Flags:	pm-rhel: mirror+

Attachments	(Terms of Use)
os_tests.tests.test_general_check.TestGeneralCheck.test_check_journalctl_dumpedcore.debug (616.11 KB, text/plain) 2021-11-30 12:25 UTC, Frank Liang	no flags	Details
Reproducer script (2.39 KB, text/x-python3) 2022-03-11 11:40 UTC, Thomas Haller	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-104280	None	None	None	2021-11-30 12:32:23 UTC
Red Hat Product Errata	RHBA-2022:8265	None	None	None	2022-11-15 10:49:51 UTC
freedesktop.org Gitlab	NetworkManager/NetworkManager/-/commit/549761b0adfc0ce1ceb1fd101ae458e1720f23b6	None	None	None	2022-03-09 14:33:03 UTC

Description Frank Liang 2021-11-30 12:25:53 UTC

Created attachment 1844175 [details]
os_tests.tests.test_general_check.TestGeneralCheck.test_check_journalctl_dumpedcore.debug

Description of problem:

When ran tests(os-tests or avocado-cloud) on aws with nm-cloud-setup enabled, the process crashed randomly.

Below is part of logs:
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: Finished Network Manager Wait Online.
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-wait-online comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal nm-cloud-setup[669]: GError set over the top of a previous GError or uninitialized memory.
                                                                               This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
                                                                               The overwriting error message was: Request cancelled
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal kernel: show_signal_msg: 31 callbacks suppressed
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal kernel: nm-cloud-setup[669]: segfault at 7ffe00000002 ip 00007fa83e6116cf sp 00007ffe035be4a0 error 4 in libglib-2.0.so.0.6800.4[7fa83e5ee000+90000]
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal kernel: Code: 25 28 00 00 00 48 89 84 24 b8 00 00 00 31 c0 48 85 ff 0f 84 f3 01 00 00 4c 8d 25 ac 82 0f 00 48 89 fb 4c 89 e7 e8 71 41 06 00 <8b> 33 48 8b 3d 68 82 0f 00 e8 e3 89 ff ff 48 85 c0 0f 84 ca 00 00
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal audit[669]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=669 comm="nm-cloud-setup" exe="/usr/libexec/nm-cloud-setup" sig=11 res=1
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: Created slice Slice /system/systemd-coredump.
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal audit: BPF prog-id=33 op=LOAD
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal audit: BPF prog-id=34 op=LOAD
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal audit: BPF prog-id=35 op=LOAD
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: Started Process Core Dump (PID 712/UID 0).
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-712-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd-coredump[713]: Process 669 (nm-cloud-setup) of user 0 dumped core.
                                                                                 
                                                                                 Found module linux-vdso.so.1 with build-id: dcfc4401f3455bb2e630438dcb294cda38a4f49e
                                                                                 Found module libcrypt.so.2 with build-id: 6ce4e5eb200e61d07398af52f8bcb316cf8466e0
                                                                                 Found module libbrotlicommon.so.1 with build-id: 4484908208e1f3ccfc9b4fb0a7dccdb8ba31e461
                                                                                 Found module libsasl2.so.3 with build-id: 2864d8e1600f3b46016fd62c44269e92c5ec0ca1
                                                                                 Found module libresolv.so.2 with build-id: caee197be64e2427319b4b89aa4954841b4cfd7b
                                                                                 Found module libkeyutils.so.1 with build-id: 83c6539bd0d3140678ba836b8baa1b215efa2632
                                                                                 Found module libkrb5support.so.0 with build-id: 1f50514af2c30a0ca5f58b17fcdeafb487f31641
                                                                                 Found module libpcre2-8.so.0 with build-id: a5dafbac0243bdc2e3b5ab9a07d61d4b32f86927
                                                                                 Found module libblkid.so.1 with build-id: 25ffb35e40228ef3df3249c7f1d8cd6135cbcf08
                                                                                 Found module libgmp.so.10 with build-id: 754b87178e725c166a68353f043ba7638e0723e1
                                                                                 Found module libhogweed.so.6 with build-id: 21c32760f1f50b9d69b43e966b93fb3de8baa076
                                                                                 Found module libnettle.so.8 with build-id: 3eda844d5a5df2443a281158b0d7ca6e81483b59
                                                                                 Found module libtasn1.so.6 with build-id: 3d3a2f6f0d4a70919496afe25e329abd189b7882
                                                                                 Found module libunistring.so.2 with build-id: 15e34cdfafa3547f9c700489b842ceb86f6fb73e
                                                                                 Found module libp11-kit.so.0 with build-id: b03163bf5b98313c7347e7270627a6b2dcd35060
                                                                                 Found module libbrotlidec.so.1 with build-id: 753ad737d1a65860a1859053e42f74288043f9cc
                                                                                 Found module liblber-2.4.so.2 with build-id: 3e4195ce506370078b6430a71bd600ed4137cd63
                                                                                 Found module libldap_r-2.4.so.2 with build-id: a681ea7664d1301d7fa11e7f092af3bf34ca835c
                                                                                 Found module libcom_err.so.2 with build-id: deb039f83bae82e65be49dd97ac3d900a053e265
                                                                                 Found module libk5crypto.so.3 with build-id: e87ad9b36510147f769a6fce54666a45072a3760
                                                                                 Found module libkrb5.so.3 with build-id: 966d55a975b9ab4ef50902036e16956964e353fb
                                                                                 Found module libgssapi_krb5.so.2 with build-id: b4a643e57e50cfbf606baddf84d4c27c65030a99
                                                                                 Found module libcrypto.so.3 with build-id: 7e9146bf80f24f7aa4bf6ef3f8b1b077a4978a01
                                                                                 Found module libssl.so.3 with build-id: 65ef1a271703a8f79f840bdef6d6699e85172adb
                                                                                 Found module libpsl.so.5 with build-id: a6b7ff002e3047642554bc414c2fca2f4c1c8f0d
                                                                                 Found module libssh.so.4 with build-id: 9ede25043bbc83dcf511ea91f76faafd11c04c77
                                                                                 Found module libidn2.so.0 with build-id: 958c50fc94ecb196b24f3619762e7ec3f28a5b40
                                                                                 Found module libnghttp2.so.14 with build-id: 7eadbd56a0e5bcd3d8a6b39b9bab2327e380283a
                                                                                 Found module libpcre.so.1 with build-id: cffb947bcc416dca3cd249cdb0a1c6f614549c30
                                                                                 Found module libffi.so.8 with build-id: 48e3675db4765a2e42729140922e11a10016f7ab
                                                                                 Found module libselinux.so.1 with build-id: c81460dd642ba18f25585fd3dc315d4d00b28c2b
                                                                                 Found module libmount.so.1 with build-id: 1077c04e3ad4a2c41847935efac0dedefe8dab78
                                                                                 Found module libz.so.1 with build-id: 62196acf9f75625acb09840435bb396c62829e12
                                                                                 Found module libudev.so.1 with build-id: b75395464a44901277d906532ab3c966a165e95d
                                                                                 Found module libgmodule-2.0.so.0 with build-id: ab90cbb51f481af02398bdfec1b78bdd50aae24c
                                                                                 Found module libgnutls.so.30 with build-id: 85df255c282b3d7f3b3b767ac3a45d0aef01b32f
                                                                                 Found module ld-linux-x86-64.so.2 with build-id: 7eb145645b4a093bf0254b5728acf9baa2a553a3
                                                                                 Found module libc.so.6 with build-id: 3ac9d4d3d020d35647834eb325eedcb9daca1fac
                                                                                 Found module libgcc_s.so.1 with build-id: 8b459c165f153aa7aedfe0c5c16955b9e9ff9320
                                                                                 Found module libcurl.so.4 with build-id: 14e25e22696e62d771934565049f822a0d01e948
                                                                                 Found module libglib-2.0.so.0 with build-id: 73abd62c90f2f2f1a79c01493416f71233396e80
                                                                                 Found module libgobject-2.0.so.0 with build-id: 7fb4384bfb09e8637fd017a91e2d178312dfe1d4
                                                                                 Found module libgio-2.0.so.0 with build-id: 9391c31fbd837e548c45f539c2fe38c2e97dcdb3
                                                                                 Found module libnm.so.0 with build-id: 9c0f4d1e4fb0ea9c380b2764d5cdd1b4c81b1b9c
                                                                                 Found module nm-cloud-setup with build-id: 9144cdf7e425ee54682a4876abcd07cabdfeb784
                                                                                 Stack trace of thread 669:
                                                                                 #0  0x00007fa83e6116cf g_error_free (libglib-2.0.so.0 + 0x406cf)
                                                                                 #1  0x00007fa83e813f2d g_task_finalize (libgio-2.0.so.0 + 0xadf2d)
                                                                                 #2  0x00007fa83e72bb28 g_object_unref (libgobject-2.0.so.0 + 0x20b28)
                                                                                 #3  0x00007fa83e6203f3 g_source_callback_unref (libglib-2.0.so.0 + 0x4f3f3)
                                                                                 #4  0x00007fa83e62451c g_source_destroy_internal (libglib-2.0.so.0 + 0x5351c)
                                                                                 #5  0x00007fa83e6260e0 g_main_context_dispatch (libglib-2.0.so.0 + 0x550e0)
                                                                                 #6  0x00007fa83e67a608 g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa9608)
                                                                                 #7  0x00007fa83e625563 g_main_loop_run (libglib-2.0.so.0 + 0x54563)
                                                                                 #8  0x000055ffbde2a904 nmc_client_new_waitsync.constprop.0 (nm-cloud-setup + 0xa904)
                                                                                 #9  0x000055ffbde277ac main (nm-cloud-setup + 0x77ac)
                                                                                 #10 0x00007fa83e336560 __libc_start_call_main (libc.so.6 + 0x2d560)
                                                                                 #11 0x00007fa83e33660c __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d60c)
                                                                                 #12 0x000055ffbde29415 _start (nm-cloud-setup + 0x9415)
                                                                                 
                                                                                 Stack trace of thread 697:
                                                                                 #0  0x00007fa83e40e97f __poll (libc.so.6 + 0x10597f)
                                                                                 #1  0x00007fa83e67a59c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa959c)
                                                                                 #2  0x00007fa83e625563 g_main_loop_run (libglib-2.0.so.0 + 0x54563)
                                                                                 #3  0x00007fa83e8775ea gdbus_shared_thread_func.lto_priv.0 (libgio-2.0.so.0 + 0x1115ea)
                                                                                 #4  0x00007fa83e654662 g_thread_proxy (libglib-2.0.so.0 + 0x83662)
                                                                                 #5  0x00007fa83e396af7 start_thread (libc.so.6 + 0x8daf7)
                                                                                 #6  0x00007fa83e41b830 __clone3 (libc.so.6 + 0x112830)
                                                                                 
                                                                                 Stack trace of thread 695:
                                                                                 #0  0x00007fa83e40e97f __poll (libc.so.6 + 0x10597f)
                                                                                 #1  0x00007fa83e67a59c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa959c)
                                                                                 #2  0x00007fa83e6236d3 g_main_context_iteration (libglib-2.0.so.0 + 0x526d3)
                                                                                 #3  0x00007fa83e623721 glib_worker_main (libglib-2.0.so.0 + 0x52721)
                                                                                 #4  0x00007fa83e654662 g_thread_proxy (libglib-2.0.so.0 + 0x83662)
                                                                                 #5  0x00007fa83e396af7 start_thread (libc.so.6 + 0x8daf7)
                                                                                 #6  0x00007fa83e41b830 __clone3 (libc.so.6 + 0x112830)
                                                                                 
                                                                                 Stack trace of thread 696:
                                                                                 #0  0x00007fa83e4140cd syscall (libc.so.6 + 0x10b0cd)
                                                                                 #1  0x00007fa83e67428c g_cond_wait_until (libglib-2.0.so.0 + 0xa328c)
                                                                                 #2  0x00007fa83e5f6481 g_async_queue_pop_intern_unlocked (libglib-2.0.so.0 + 0x25481)
                                                                                 #3  0x00007fa83e6576aa g_thread_pool_thread_proxy.lto_priv.0 (libglib-2.0.so.0 + 0x866aa)
                                                                                 #4  0x00007fa83e654662 g_thread_proxy (libglib-2.0.so.0 + 0x83662)
                                                                                 #5  0x00007fa83e396af7 start_thread (libc.so.6 + 0x8daf7)
                                                                                 #6  0x00007fa83e41b830 __clone3 (libc.so.6 + 0x112830)
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: systemd-coredump: Deactivated successfully.
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-712-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: nm-cloud-setup.service: Main process exited, code=dumped, status=11/SEGV
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: nm-cloud-setup.service: Failed with result 'core-dump'.
Nov 29 03:41:56 ip-10-116-2-68.us-west-2.compute.internal systemd[1]: Stopped Automatically configure NetworkManager in cloud.

RHEL Version:
RHEL-9.0(5.14.0-1.7.1.el9.x86_64)

How reproducible:
100%

Steps to Reproduce:
1. start a rhel-9 beta  instance
2. install os-tests via pip3
3. run os-tests
# os-tests --host <hostname> --user ec2-user --keyfile <ssh_key> --result <logdir> 

Actual results:
nm-cloud-setup crash log

Expected results:
no crash found

Additional info:
- N/A

Comment 1 Gris Ge 2021-12-08 07:55:05 UTC

Hi Frank,

Could you rerun your test against NetworkManager-1.36.0-0.2.el9 ?

Thank you!

Comment 2 Frank Liang 2021-12-14 06:59:14 UTC

(In reply to Gris Ge from comment #1)
> Hi Frank,
> 
> Could you rerun your test against NetworkManager-1.36.0-0.2.el9 ?
I did not reproduce it in RHEL-9 nightly compose with NetworkManager-1.36.0-0.2.el9.x86_64.
But seems it is a random issue that I found 2 failures in our ci job with NetworkManager-1.36.0-0.2.el8.aarch64 installed.

Comment 7 Wen Liang 2022-03-09 14:31:55 UTC

The crash was fixed in the upstream, https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/549761b0adfc0ce1ceb1fd101ae458e1720f23b6

Comment 10 Thomas Haller 2022-03-11 11:40:04 UTC

Created attachment 1865434 [details]
Reproducer script

Reproducer script for the crash.


It shows that Wen's patch [1] fixes the most prominent problem.

There is a related bug (fixed by commit [2]), which the reproducer script also shows. Patch [2] is strongly related to this rhbz, however it only gets triggered in very special circumstances (when setting NM_CLIENT_DBUS_CONNECTION *and* cancel the cancellable before calling init_async()) -- which nm-cloud-setup doesn't do.

Finally, the script also uncovers another bug (with debug-builds --with-more-asserts). Fixed by [3]. So to run the script successfully, you need all three patches. Or uncomment the line `# use_prepared_dbus_connection = False` (then you cannot reproduce [2] and [3] issue).

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/549761b0adfc0ce1ceb1fd101ae458e1720f23b6
[2] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/a9d521bf8c183b9bb95375545ac514de170e3484
[3] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/ae0cc9618c49bb74bbe54a073dc337e9a3b0005b

Comment 13 Thomas Haller 2022-03-14 14:45:53 UTC

adding test to CI: https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/992

Comment 15 errata-xmlrpc 2022-11-15 10:49:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8265

Note You need to log in before you can comment on or make changes to this bug.