Bug 1494065 - [NMCI][abrt] [faf] NetworkManager: unknown function(): /usr/bin/nmcli killed by 6
[NMCI][abrt] [faf] NetworkManager: unknown function(): /usr/bin/nmcli killed ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: glib2 (Show other bugs)
7.4
Unspecified Unspecified
unspecified Severity high
: rc
: ---
Assigned To: Colin Walters
Desktop QE
http://faf.lab.eng.brq.redhat.com/faf...
:
: 1450075 1512484 1513368 1525197 (view as bug list)
Depends On:
Blocks: 1470965
  Show dependency treegraph
 
Reported: 2017-09-21 08:25 EDT by Vladimir Benes
Modified: 2018-05-17 10:39 EDT (History)
16 users (show)

See Also:
Fixed In Version: glib2-2.54.2-2.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-10 09:04:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
backtrace (27.38 KB, text/plain)
2017-09-28 09:44 EDT, Beniamino Galvani
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
GNOME Bugzilla 788368 None None None 2017-09-30 05:40 EDT
Red Hat Product Errata RHBA-2018:0770 None None None 2018-04-10 09:05 EDT

  None (edit)
Description Vladimir Benes 2017-09-21 08:25:32 EDT
This bug has been created based on an anonymous crash report requested by the package maintainer.

Report URL: http://faf.lab.eng.brq.redhat.com/faf/reports/bthash/5075cbf20dbdf2a172171977eb38a9d3c3ca4e6c/
Comment 1 Vladimir Benes 2017-09-21 08:27:03 EDT
as Beniamino pointed out there may be some interesting details here:
 
"but the backtrace looks interesting because there are 2 parallel execution of add_interfaces() and this may be an hint to the actual problem...."
Comment 2 Beniamino Galvani 2017-09-28 09:43:01 EDT
We see frequent crashes of nmcli in the GLib code when the object
manager client is initialized. For example:

Thread 1:
#0  0x00007f848fed0c15 in _int_malloc (av=av@entry=0x7f8470000020, bytes=bytes@entry=128) at malloc.c:3782
#1  0x00007f848fed310c in __GI___libc_malloc (bytes=128) at malloc.c:2897
#2  0x00007f849067fb66 in g_realloc (mem=0x0, n_bytes=128) at gmem.c:159
#3  0x00007f849064e07a in g_ptr_array_maybe_expand (array=array@entry=0x7f847800b4e0, len=len@entry=1) at garray.c:1118
#4  0x00007f849064f18b in g_ptr_array_add (array=0x7f847800b4e0, data=0x7f847000aec0) at garray.c:1382
#5  0x00007f849067a198 in g_main_context_check (context=context@entry=0x7f847000ad60, max_priority=0, fds=fds@entry=0x7f847000afb0, n_fds=n_fds@entry=1) at gmain.c:3817
#6  0x00007f849067a730 in g_main_context_iterate (context=0x7f847000ad60, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3924
#7  0x00007f849067aaea in g_main_loop_run (loop=0x7f847000ac60) at gmain.c:4123
#8  0x00007f8490c5c23b in initable_init (initable=0x7f8470009a50, cancellable=0x0, error=0x7f8486763a98) at gdbusproxy.c:1949
#9  0x00007f8490bf31fa in g_initable_new_valist (object_type=<optimized out>, first_property_name=first_property_name@entry=0x7f8490ccaf90 "g-connection", var_args=var_args@entry=0x7f8486763910, cancellable=cancellable@entry=0x0, error=error@entry=0x7f8486763a98) at ginitable.c:228
#10 0x00007f8490bf32e9 in g_initable_new (object_type=object_type@entry=94165753751728, cancellable=cancellable@entry=0x0, error=error@entry=0x7f8486763a98, first_property_name=first_property_name@entry=0x7f8490ccaf90 "g-connection") at ginitable.c:146
#11 0x00007f8490c69d42 in add_interfaces (manager=manager@entry=0x55a4ac4e3460 [GDBusObjectManagerClient], object_path=0x7f847c06d5e0 "/org/freedesktop/NetworkManager/IP4Config/75", ifaces_and_properties=<optimized out>, name_owner=name_owner@entry=0x7f8478019310 ":1.1680") at gdbusobjectmanagerclient.c:1495
#12 0x00007f8490c6a954 in process_get_all_result (manager=0x55a4ac4e3460 [GDBusObjectManagerClient], value=<optimized out>, name_owner=0x7f8478019310 ":1.1680") at gdbusobjectmanagerclient.c:1647
#13 0x00007f8490c6b0c3 in initable_init (initable=0x55a4ac4e3460, cancellable=0x0, error=0x7f8486763d20) at gdbusobjectmanagerclient.c:1400
#14 0x00007f8490bcb0de in async_init_thread (task=0x55a4ac4e9840 [GTask], source_object=<optimized out>, task_data=<optimized out>, cancellable=<optimized out>) at gasyncinitable.c:257
...

Thread 2:
#0  0x00007f848fe881f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f848fe898e8 in __GI_abort () at abort.c:90
#2  0x00007f848fec7f47 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f848ffd4608 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007f848fecf619 in _int_free (ar_ptr=0x7f8478000020, ptr=<optimized out>, str=0x7f848ffd46a0 "free(): invalid next size (normal)", action=3) at malloc.c:5023
#4  0x00007f848fecf619 in _int_free (av=0x7f8478000020, p=<optimized out>, have_lock=0) at malloc.c:3845
#5  0x00007f849067fbce in g_free (mem=0x7f847801ca30) at gmem.c:189
#6  0x00007f84906689e1 in g_hash_table_resize (hash_table=hash_table@entry=0x55a4ac4cb580 = {...}) at ghash.c:619
#7  0x00007f8490668fe4 in g_hash_table_insert_node (hash_table=0x55a4ac4cb580 = {...}) at ghash.c:645
#8  0x00007f8490668fe4 in g_hash_table_insert_node (hash_table=hash_table@entry=0x55a4ac4cb580 = {...}, node_index=node_index@entry=4, key_hash=key_hash@entry=3982537675, new_key=new_key@entry=0x55a4ac4e97c0, new_value=new_value@entry=0x55a4ac4e2160, keep_new_key=keep_new_key@entry=0, reusing_key=reusing_key@entry=0) at ghash.c:978
#9  0x00007f84906690e7 in g_hash_table_insert_internal (hash_table=0x55a4ac4cb580 = {...}, key=0x55a4ac4e97c0, value=0x55a4ac4e2160, keep_new_key=0) at ghash.c:1229
#10 0x00007f8490c69f4f in add_interfaces (manager=manager@entry=0x55a4ac4e3460 [GDBusObjectManagerClient], object_path=0x55a4ac4e9640 "/org/freedesktop/NetworkManager/IP6Config/85", ifaces_and_properties=<optimized out>, name_owner=0x7f8478019310 ":1.1680") at gdbusobjectmanagerclient.c:1558
#11 0x00007f8490c6a80b in on_control_proxy_g_signal (proxy=<optimized out>, sender_name=<optimized out>, signal_name=0x55a4ac4e7bb0 "InterfacesAdded", parameters=0x7f8478009070, user_data=0x55a4ac4e3460) at gdbusobjectmanagerclient.c:1672
#12 0x00007f848ea2edcc in ffi_call_unix64 () at ../src/x86/unix64.S:76
#13 0x00007f848ea2e6f5 in ffi_call (cif=cif@entry=0x7ffd3f42aea0, fn=<optimized out>, rvalue=0x7ffd3f42ae10, avalue=avalue@entry=0x7ffd3f42ad90) at ../src/x86/ffi64.c:522
#18 0x00007f849096bddf in <emit signal ??? on instance 0x7f8478004400 [GDBusProxy]> (instance=instance@entry=0x7f8478004400, signal_id=<optimized out>, detail=detail@entry=0) at gsignal.c:3447
    #14 0x00007f84909521a8 in g_cclosure_marshal_generic (closure=0x7f8478019f70, return_gvalue=0x0, n_param_values=<optimized out>, param_values=<optimized out>, invocation_hint=<optimized out>, marshal_data=0x0) at gclosure.c:1490
    #15 0x00007f8490951968 in g_closure_invoke (closure=0x7f8478019f70, return_value=return_value@entry=0x0, n_param_values=4, param_values=param_values@entry=0x7ffd3f42b0a0, invocation_hint=invocation_hint@entry=0x7ffd3f42b040) at gclosure.c:804
    #16 0x00007f8490963a7d in signal_emit_unlocked_R (node=node@entry=0x7f8478018290, detail=detail@entry=0, instance=instance@entry=0x7f8478004400, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x7ffd3f42b0a0) at gsignal.c:3635
    #17 0x00007f849096baf1 in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>, var_args=var_args@entry=0x7ffd3f42b260) at gsignal.c:3391
#19 0x00007f8490c5ae7c in on_signal_received (connection=<optimized out>, sender_name=0x7f847c0721f0 ":1.1680", object_path=<optimized out>, interface_name=<optimized out>, signal_name=0x7f847c0722a0 "InterfacesAdded", parameters=0x7f8478009070, user_data=0x7f8478015d60) at gdbusproxy.c:917
#20 0x00007f8490c4a7b5 in emit_signal_instance_in_idle_cb (data=data@entry=0x7f847c074950) at gdbusconnection.c:3705
#21 0x00007f84906770d7 in g_idle_dispatch (source=0x7f847c04d0d0, callback=0x7f8490c4a740 <emit_signal_instance_in_idle_cb>, user_data=0x7f847c074950) at gmain.c:5543
...

I'm attaching the full backtrace.

It seems to me that the call to:

  g_hash_table_insert (manager->priv->map_object_path_to_object_proxy, ...)

in add_interfaces() in not protected by locks and can race with other
operations on the hash table in other threads.

Reassigning the bug to glib.
Comment 3 Beniamino Galvani 2017-09-28 09:44 EDT
Created attachment 1331993 [details]
backtrace
Comment 4 Beniamino Galvani 2017-10-11 11:50:12 EDT
Fixed upstream:

https://git.gnome.org/browse/glib/commit/?id=aeecd81dd13e4d8ef609149f82770ad06a8fccdb

Since we're seeing the crash quite often in nmcli, can the fix be included in RHEL 7.5?
Comment 5 Beniamino Galvani 2017-11-07 08:07:02 EST
Ping, any news on this? We continue to see crashes caused by this bug in our CI tests.
Comment 6 Thomas Haller 2017-11-07 08:53:36 EST
*** Bug 1450075 has been marked as a duplicate of this bug. ***
Comment 7 Colin Walters 2017-11-08 10:18:43 EST
Does anyone know what incantations need to be made for this to get pm_ack+?
Comment 8 Tomas Pelka 2017-11-09 09:20:35 EST
(In reply to Colin Walters from comment #7)
> Does anyone know what incantations need to be made for this to get pm_ack+?

Devel phase is officially over based on schedule so bugbot do not pm_ack automagicaly. You need to ask bgollahe.
Comment 9 Kalev Lember 2017-11-10 07:37:49 EST
I backported the patch to glib2-2.54.2-2.el7
Comment 11 Thomas Haller 2017-11-13 07:15:32 EST
*** Bug 1512484 has been marked as a duplicate of this bug. ***
Comment 12 Beniamino Galvani 2018-01-16 16:10:32 EST
*** Bug 1513368 has been marked as a duplicate of this bug. ***
Comment 16 Beniamino Galvani 2018-03-24 05:50:14 EDT
*** Bug 1525197 has been marked as a duplicate of this bug. ***
Comment 18 errata-xmlrpc 2018-04-10 09:04:05 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0770

Note You need to log in before you can comment on or make changes to this bug.