Bug 1260920 - [abrt] [beaker] [master] Creating pipes for GWakeup: Too many open files
[abrt] [beaker] [master] Creating pipes for GWakeup: Too many open files
Status: CLOSED NEXTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager (Show other bugs)
7.3
x86_64 Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Beniamino Galvani
Desktop QE
abrt_hash:0fd8057911309c9afeee68a75b0...
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-08 04:44 EDT by Lubomir Rintel
Modified: 2015-09-30 06:02 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-30 06:02:57 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: backtrace (29.49 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: cgroup (199 bytes, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: core_backtrace (10.20 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: dso_list (5.81 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: environ (72 bytes, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: limits (1.29 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: machineid (135 bytes, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: maps (26.92 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: open_fds (54.50 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: proc_pid_status (1.04 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
File: var_log_messages (43.81 KB, text/plain)
2015-09-08 04:44 EDT, Lubomir Rintel
no flags Details
[PATCH] core: fix file descriptor leak in bus manager (1.09 KB, patch)
2015-09-08 10:37 EDT, Beniamino Galvani
no flags Details | Diff
[PATCH v2] core: fix file descriptor leak in bus manager (1.20 KB, patch)
2015-09-08 12:39 EDT, Beniamino Galvani
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
GNOME Bugzilla 754730 None None None Never

  None (edit)
Description Lubomir Rintel 2015-09-08 04:44:27 EDT
Description of problem:
Sep 07 16:16:02 myown.hostname NetworkManager[6845]: <debug> [1441656962.986592] [nm-exported-object.c:571] idle_emit_properties_changed(): PropertiesChanged NMSettings 0x7fabf5c4a160: {'Connections': <[objectpath '/org/freedesktop/NetworkManager/Settings/113', '/org/freedesktop/NetworkManager/Settings/114', '/org/freedesktop/NetworkManager/Settings/115', '/org/freedesktop/NetworkManager/Settings/116', '/org/freedesktop/NetworkManager/Settings/117', '/org/freedesktop/NetworkManager/Settings/118', '/org/freedesktop/NetworkManager/Settings/119', '/org/freedesktop/NetworkManager/Settings/120', '/org/freedesktop/NetworkManager/Settings/10', '/org/freedesktop/NetworkManager/Settings/11', '/org/freedesktop/NetworkManager/Settings/12', '/org/freedesktop/NetworkManager/Settings/13', '/org/freedesktop/NetworkManager/Settings/0', '/org/freedesktop/NetworkManager/Settings/1', '/org/freedesktop/NetworkManager/Settings/2', '/org/freedesktop/NetworkManager/Settings/3', '/org/freedesktop/NetworkManager/Settings/4', '/org/freedesktop/NetworkManager/Settings/5', '/org/freedesktop/NetworkManager/Settings/6', '/org/freedesktop/NetworkManager/Settings/7', '/org/freedesktop/NetworkManager/Settings/8', '/org/freedesktop/NetworkManager/Settings/9']>, 'Connections': <[objectpath '/org/freedesktop/NetworkManager/Settings/114', '/org/freedesktop/NetworkManager/Settings/115', '/org/freedesktop/NetworkManager/Settings/116', '/org/freedesktop/NetworkManager/Settings/117', '/org/freedesktop/NetworkManager/Settings/118', '/org/freedesktop/NetworkManager/Settings/119', '/org/freedesktop/NetworkManager/Settings/120', '/org/freedesktop/NetworkManager/Settings/10', '/org/freedesktop/NetworkManager/Settings/11', '/org/freedesktop/NetworkManager/Settings/12', '/org/freedesktop/NetworkManager/Settings/13', '/org/freedesktop/NetworkManager/Settings/0', '/org/freedesktop/NetworkManager/Settings/1', '/org/freedesktop/NetworkManager/Settings/2', '/org/freedesktop/NetworkManager/Settings/3', '/org/freedesktop/NetworkManager/Settings/4', '/org/freedesktop/NetworkManager/Settings/5', '/org/freedesktop/NetworkManager/Settings/6', '/org/freedesktop/NetworkManager/Settings/7', '/org/freedesktop/NetworkManager/Settings/8', '/org/freedesktop/NetworkManager/Settings/9']>}
Sep 07 16:16:02 myown.hostname NetworkManager[6845]: <debug> [1441656962.995517] [nm-exported-object.c:571] idle_emit_properties_changed(): PropertiesChanged NMDeviceEthernet 0x7fabf5d09eb0: {'AvailableConnections': <[objectpath '/org/freedesktop/NetworkManager/Settings/11']>}
Sep 07 16:16:03 myown.hostname NetworkManager[6845]: <debug> [1441656963.004245] [nm-bus-manager.c:126] private_server_closed(): (private) closed connection 0x7fabd413d900 on private socket.
Sep 07 16:16:03 myown.hostname sudo[24983]:     root : TTY=unknown ; PWD=/mnt/tests/NetworkManager ; USER=root ; COMMAND=/sbin/ip link del bridge0
Sep 07 16:16:03 myown.hostname sudo[24989]:     root : TTY=unknown ; PWD=/mnt/tests/NetworkManager ; USER=root ; COMMAND=/bin/nmcli con delete bridge-slave-eth3 bridge-slave-eth4 bridge-slave-eth5 bridge-slave-eth6 bridge-slave-eth7 bridge-slave-eth8 bridge-slave-eth9
Sep 07 16:16:03 myown.hostname NetworkManager[6845]: <debug> [1441656963.060551] [nm-bus-manager.c:153] private_server_new_connection(): (private) accepted connection 0x7fabd413da00 on private socket.
Sep 07 16:16:03 myown.hostname NetworkManager[6845]: <debug> [1441656963.170911] [nm-bus-manager.c:126] private_server_closed(): (private) closed connection 0x7fabd413da00 on private socket.
Sep 07 16:16:04 myown.hostname NetworkManager[6845]: <debug> [1441656964.465308] [platform/nm-linux-platform.c:2344] event_notification(): platform-linux: event-notification: NEWROUTE, seq 0: 2620:52:0:1040:5054:ff:fe38:5099/128 via fe80:52:0:1086::fc dev 2 metric 0 mss 0 src rtm-f-cloned
Sep 07 16:16:05 myown.hostname nmtui/./runtest.sh[25026]: Test nmtui_bridge_add_many_slaves finished with result PASS: 0
Sep 07 16:16:08 myown.hostname nmtui/./runtest.sh[25054]: Running test nmtui_bridge_over_ethernet_devices
Sep 07 16:16:09 myown.hostname sudo[25063]:     root : TTY=unknown ; PWD=/mnt/tests/NetworkManager ; USER=root ; COMMAND=/bin/pkill nmtui
Sep 07 16:16:10 myown.hostname NetworkManager[6845]: (NetworkManager:6845): GLib-ERROR **: Creating pipes for GWakeup: Too many open files
Sep 07 16:16:10 myown.hostname kernel: traps: NetworkManager[6845] trap int3 ip:7fabf139d8c3 sp:7ffe54704f70 error:0
Sep 07 16:16:10 myown.hostname systemd[1]: NetworkManager.service: main process exited, code=dumped, status=5/TRAP
Sep 07 16:16:10 myown.hostname systemd[1]: Unit NetworkManager.service entered failed state.
Sep 07 16:16:10 myown.hostname systemd[1]: NetworkManager.service failed.

Version-Release number of selected component:
NetworkManager-1.1.0-12937.e6c64af8be.el7.centos

Additional info:
reporter:       libreport-2.1.11
backtrace_rating: 4
cmdline:        /usr/sbin/NetworkManager --no-daemon
crash_function: g_wakeup_new
executable:     /usr/sbin/NetworkManager
kernel:         3.10.0-302.el7.x86_64
runlevel:       N 3
type:           CCpp
uid:            0

Truncated backtrace:
Thread no. 1 (6 frames)
 #2 g_wakeup_new at gwakeup.c:161
 #3 g_cancellable_make_pollfd at gcancellable.c:405
 #4 g_socket_condition_timed_wait at gsocket.c:3650
 #5 g_socket_accept at gsocket.c:2234
 #6 accept_ready at gsocketlistener.c:696
 #7 socket_source_dispatch at gsocket.c:3264
Comment 1 Lubomir Rintel 2015-09-08 04:44:30 EDT
Created attachment 1071241 [details]
File: backtrace
Comment 2 Lubomir Rintel 2015-09-08 04:44:31 EDT
Created attachment 1071242 [details]
File: cgroup
Comment 3 Lubomir Rintel 2015-09-08 04:44:32 EDT
Created attachment 1071243 [details]
File: core_backtrace
Comment 4 Lubomir Rintel 2015-09-08 04:44:33 EDT
Created attachment 1071244 [details]
File: dso_list
Comment 5 Lubomir Rintel 2015-09-08 04:44:34 EDT
Created attachment 1071245 [details]
File: environ
Comment 6 Lubomir Rintel 2015-09-08 04:44:35 EDT
Created attachment 1071246 [details]
File: limits
Comment 7 Lubomir Rintel 2015-09-08 04:44:36 EDT
Created attachment 1071247 [details]
File: machineid
Comment 8 Lubomir Rintel 2015-09-08 04:44:37 EDT
Created attachment 1071248 [details]
File: maps
Comment 9 Lubomir Rintel 2015-09-08 04:44:38 EDT
Created attachment 1071249 [details]
File: open_fds
Comment 10 Lubomir Rintel 2015-09-08 04:44:39 EDT
Created attachment 1071250 [details]
File: proc_pid_status
Comment 11 Lubomir Rintel 2015-09-08 04:44:40 EDT
Created attachment 1071251 [details]
File: var_log_messages
Comment 13 Beniamino Galvani 2015-09-08 10:37:02 EDT
Created attachment 1071384 [details]
[PATCH] core: fix file descriptor leak in bus manager

The attached patch resolves the problem, however I'm not quite sure of the fix. Can someone please have a look at it?

Steps to reproduce:

1. lsof -p `pidof NetworkManager`
2. activate a connection with ipv4.method = auto
3. lsof -p `pidof NetworkManager` # check if there are additional '/var/run/NetworkManager/private' unix sockets
Comment 14 Thomas Haller 2015-09-08 10:54:07 EDT
(In reply to Beniamino Galvani from comment #13)
> Created attachment 1071384 [details]
> [PATCH] core: fix file descriptor leak in bus manager
> 
> The attached patch resolves the problem, however I'm not quite sure of the
> fix. Can someone please have a look at it?

I don't know, but it doesn't look wrong (if it fixes the problem).

Is it possible that we leak the GDBusConnection? (you could find out by attaching a weak-ref and logging it's destruction).
Because it seems to me that destroying the connection should also close the file handle...

CC-ing danw.
Comment 15 Dan Winship 2015-09-08 11:15:57 EDT
I think this is a bug in GLib. In particular, if you change that to g_dbus_connection_close_sync(), then it will leak the fd in this case just like we were doing before.

It might be better to put this in private_server_closed(), with a FIXME comment?
Comment 16 Beniamino Galvani 2015-09-08 12:39:35 EDT
Created attachment 1071412 [details]
[PATCH v2] core: fix file descriptor leak in bus manager

(In reply to Thomas Haller from comment #14)
> Is it possible that we leak the GDBusConnection? (you could find out by
> attaching a weak-ref and logging it's destruction).

Apparently the connection is not leaked, but a close() is required to free
the connection resources.

(In reply to Dan Winship from comment #15)
> I think this is a bug in GLib. In particular, if you change that to
> g_dbus_connection_close_sync(), then it will leak the fd in this case just
> like we were doing before.
>
> It might be better to put this in private_server_closed(), with a FIXME
> comment?

What about the attached patch?
Comment 17 Dan Winship 2015-09-08 14:20:28 EDT
lgtm
Comment 18 Beniamino Galvani 2015-09-09 04:19:38 EDT
Changed comment style to C89 and applied to master:

8318672fe997 core: fix file descriptor leak in bus manager
Comment 19 Thomas Haller 2015-09-30 06:02:57 EDT
This bug was opened for a test-failure while testing upstream master.

nm-1-0 branch is not affected because it doesn't contain the code in question (NMBusManager). Especially, there are no affected RHEL-7 builds.


Closing BZ.

Note You need to log in before you can comment on or make changes to this bug.