1189028 – docker fd leak

Bug 1189028 - docker fd leak

Summary: docker fd leak

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Fedora EPEL
Classification:	Fedora
Component:	docker-io
Sub Component:
Version:	el6
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Assignee:	Lokesh Mandvekar
QA Contact:	Fedora Extras Quality Assurance
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1210472 (view as bug list)
Depends On:
Blocks:	1202517
TreeView+	depends on / blocked

Reported:	2015-02-04 09:35 UTC by Watanabe Takashi
Modified:	2016-04-02 06:37 UTC (History)
CC List:	22 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1202517 (view as bug list)
Environment:
Last Closed:	2016-04-02 06:37:02 UTC
Type:	Bug

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Watanabe Takashi 2015-02-04 09:35:10 UTC

Description of problem:

 docker daemon doesn't close file descriptor.

Version-Release number of selected component (if applicable):
 docker-io-1.4.1-3.el6.x86_64

How reproducible:


Steps to Reproduce:
1. while true; do docker run -t -i --rm busybox /bin/true ; done
2. (in other terminal) DEBUG=1 docker info
3. check values at Fds: and Goroutines: lines

Actual results:
 Fds and Goroutines values are increased forever

Expected results:


Additional info:

- It doesn't happen on RHEL7 + docker 1.4.1
- It doesn't happen on RHEL6 + docker-io-1.3.2-2.el6.x86_64

# lsof -p $(pidof docker)
<snip>
docker  7771 root   14u   REG                0,9        0   3791 [eventfd]
docker  7771 root   15u   REG                0,9        0   3791 [eventfd]
docker  7771 root   16r   REG               0,21        0  20709 /cgroup/memory/docker/98b529e398bec4568c58ea96088aef1535eca504bed6453bab8f6f7803d31cfb/memory.oom_control (deleted)
docker  7771 root   17u   REG                0,9        0   3791 [eventfd]
docker  7771 root   18r   REG               0,21        0  21176 /cgroup/memory/docker/bc213d3e80815021780064874665fe80801c5a1010d1c0bc4bd5b118ca45d11f/memory.oom_control (deleted)
docker  7771 root   19u   REG                0,9        0   3791 [eventfd]
docker  7771 root   20r   REG               0,21        0  21626 /cgroup/memory/docker/7b8d59ec4f23c14e2732726594460ccf89fe31c06e60fabbdc4e358f794af102/memory.oom_control (deleted)
docker  7771 root   21w   REG                0,9        0   3791 [eventfd]
docker  7771 root   22r   REG               0,21        0  22112 /cgroup/memory/docker/99ac44fcb18a3e1aea00a5e7cfb040942f2c4f2c673c291105c367788e49d0f2/memory.oom_control (deleted)
docker  7771 root   24r   REG               0,21        0  22672 /cgroup/memory/docker/b1804ae48289df81d3a5c54167baf729612e12c552ef98917df90d8645d235ad/memory.oom_control (deleted)
<snip>

# ps $(pidof docker) m
  PID TTY      STAT   TIME COMMAND
 7771 pts/0    -      0:09 /usr/bin/docker -d
    - -        Sl     0:02 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:01 -
    - -        Sl     0:00 -
<snip>

Comment 1 Daniel Walsh 2015-03-09 17:25:18 UTC

Is this still a bug in docker-1.5?

Comment 2 Watanabe Takashi 2015-03-10 15:53:41 UTC

(In reply to Daniel Walsh from comment #1)
> Is this still a bug in docker-1.5?

I tried kernel-2.6.32-504.8.1.el6.x86_64, docker-io-1.5.0-1.el6.x86_64(built from http://pkgs.fedoraproject.org/cgit/docker-io.git/log/?h=el6).
This problem was reproduced.

[vagrant@localhost ~]$ uname -r
2.6.32-504.8.1.el6.x86_64
[vagrant@localhost ~]$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.3.3
Git commit (client): a8a31ef/1.5.0
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.3.3
Git commit (server): a8a31ef/1.5.0
[vagrant@localhost ~]$ for i in $(seq 10); do docker run -t -i --rm busybox /bin/true ; done
[vagrant@localhost ~]$ DEBUG=1 docker info
Containers: 0
Images: 4
Storage Driver: devicemapper
 Pool Name: docker-253:0-396434-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 311 MB
 Data Space Total: 107.4 GB
 Metadata Space Used: 823.3 kB
 Metadata Space Total: 2.147 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.8.1.el6.x86_64
Operating System: <unknown>
CPUs: 2
Total Memory: 490.2 MiB
Name: localhost.localdomain
ID: 7ZND:FDGS:G34R:CSG5:X5PZ:U27F:PX2D:PDSC:X6U5:SYSW:CHE6:IMD3
Debug mode (server): false
Debug mode (client): true
Fds: 31
Goroutines: 24
EventsListeners: 0
Init SHA1: 085c9e5c44d5bab44e695bb55892ba6852307cf5
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker
[vagrant@localhost ~]$ sudo lsof -p $(pidof docker)
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
docker  5644 root  cwd    DIR              253,0     4096      2 /
docker  5644 root  rtd    DIR              253,0     4096      2 /
docker  5644 root  txt    REG              253,0 12925591 270035 /usr/bin/docker
docker  5644 root  mem    REG              253,0    55184 130655 /lib64/libudev.so.0.5.1
docker  5644 root  mem    REG              253,0   248680 130642 /lib64/libsepol.so.1
docker  5644 root  mem    REG              253,0   122040 130643 /lib64/libselinux.so.1
docker  5644 root  mem    REG              253,0    19536 130601 /lib64/libdl-2.12.so
docker  5644 root  mem    REG              253,0  1921176 130573 /lib64/libc-2.12.so
docker  5644 root  mem    REG              253,0   287096 130611 /lib64/libdevmapper.so.1.02
docker  5644 root  mem    REG              253,0   583552 262963 /usr/lib64/libsqlite3.so.0.8.6
docker  5644 root  mem    REG              253,0   142640 130597 /lib64/libpthread-2.12.so
docker  5644 root  mem    REG              253,0   154624 130579 /lib64/ld-2.12.so
docker  5644 root    0u   CHR              136,2      0t0      5 /dev/pts/2
docker  5644 root    1w   REG              253,0   198319 396408 /var/log/docker
docker  5644 root    2w   REG              253,0   198319 396408 /var/log/docker
docker  5644 root    3r   CHR                1,9      0t0   3798 /dev/urandom
docker  5644 root    4u   CHR              10,58      0t0   5485 /dev/mapper/control
docker  5644 root    5u  unix 0xffff88001d95aa80      0t0  28908 /var/run/docker.sock
docker  5644 root    6u   REG              253,0     5120 396446 /var/lib/docker/linkgraph.db
docker  5644 root    7r   DIR               0,10        0      1 inotify
docker  5644 root    8u   REG                0,9        0   3791 [eventpoll]
docker  5644 root   14r   REG               0,21        0  29248 /cgroup/memory/docker/bd14586f8a62117ed2429c6182314be87eed3ae897c6505b41b9ec147b6ac5df/memory.oom_control (deleted)
docker  5644 root   15r   REG               0,21        0  29651 /cgroup/memory/docker/942b863aacaad28e58b2c88e9b9f445f961e4c29d727b2480fdcee09851e2755/memory.oom_control (deleted)
docker  5644 root   16u   REG                0,9        0   3791 [eventfd]
docker  5644 root   17r   REG               0,21        0  30114 /cgroup/memory/docker/1578a0311b931a960e2e9a9b43e2a8c7e593eab36796b03647aa11b99734e745/memory.oom_control (deleted)
docker  5644 root   18u   REG                0,9        0   3791 [eventfd]
docker  5644 root   19r   REG               0,21        0  30543 /cgroup/memory/docker/3afaa7caba99c43ad5f75ec73d135314d741c3ef086bf52a762317165bb2d78b/memory.oom_control (deleted)
docker  5644 root   20u   REG                0,9        0   3791 [eventfd]
docker  5644 root   21r   REG               0,21        0  30992 /cgroup/memory/docker/c472c181f7c09719b287a861a23267b93764cd5e58e8ecfe128243f81a7ff442/memory.oom_control (deleted)
docker  5644 root   22u   REG                0,9        0   3791 [eventfd]
docker  5644 root   23r   REG               0,21        0  31352 /cgroup/memory/docker/5fc09cd4d2866ddf5d64678888d6731476001e6caff2391adbccf7ca2a52654d/memory.oom_control (deleted)
docker  5644 root   24u   REG                0,9        0   3791 [eventfd]
docker  5644 root   25r   REG               0,21        0  31806 /cgroup/memory/docker/370318c2632bb34e0333ae7fea71eef55b90869374b88ab094112dbb6742c995/memory.oom_control (deleted)
docker  5644 root   26u   REG                0,9        0   3791 [eventfd]
docker  5644 root   27r   REG               0,21        0  32255 /cgroup/memory/docker/f0df7e43de3291370a96073b8cef933aeb05d78edc8c8b367462f010da5fa181/memory.oom_control (deleted)
docker  5644 root   28u   REG                0,9        0   3791 [eventfd]
docker  5644 root   29r   REG               0,21        0  32657 /cgroup/memory/docker/58a5fde3ed7c223d530e9896d587a3d90c1938f94cb3b8a72056d98cf938b94a/memory.oom_control (deleted)
docker  5644 root   30u   REG                0,9        0   3791 [eventfd]
docker  5644 root   31r   REG               0,21        0  33053 /cgroup/memory/docker/1b139ed3beecf8566aaec445d42438432c8d9e492020da26c0b8bb3dcffa4173/memory.oom_control (deleted)
docker  5644 root   32u   REG                0,9        0   3791 [eventfd]
docker  5644 root   34u   REG                0,9        0   3791 [eventfd]
[vagrant@localhost ~]$ ps m $(pidof docker)
  PID TTY      STAT   TIME COMMAND
 5644 pts/2    -      0:04 /usr/bin/docker -d
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:01 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -
    - -        Sl     0:00 -

Comment 4 Timothy St. Clair 2015-03-13 16:42:52 UTC

Is there an upstream issue around *this?

Comment 6 Timothy St. Clair 2015-03-13 17:58:57 UTC

xref - https://github.com/docker/docker/issues/8693

Comment 7 Joe Mario 2015-03-13 18:21:22 UTC

An update to my comment 5 above.
I'm now wondering if I'm hitting a different problem.

In my testing, if aio-max-nr is lower than what Oracle needs, then it triggers the file desc usage to grow rapidly till the Oracle processes eventually die/error because the system's /proc/sys/fs/file-max-nr value has been hit.  

However, once those Oracle processes exit, the file desc usage count drops accordingly.  

Joe

Comment 11 Daniel Walsh 2015-03-16 19:23:46 UTC

There is no unregistermachine.  I don't see a nice way to call conn.Close.  I have changed the code to only do conn.InitConnection once, which will at least only open one descriptor for all of the daemon run.

Comment 12 Daniel Walsh 2015-03-16 19:24:28 UTC

I have checked it into the docker-1.5.0 branch.

Comment 13 Bill C. Riemers 2015-04-09 20:40:50 UTC

*** Bug 1210472 has been marked as a duplicate of this bug. ***

Comment 14 Bill C. Riemers 2015-04-09 20:45:47 UTC

(In reply to Daniel Walsh from comment #12)
> I have checked it into the docker-1.5.0 branch.

How does that help RHEL 6 users?  The current version is:

docker-io-1.4.1-3.el6.x86_64

?

I saw a bug for swarm in github.  See the bug I just marked as a duplicate for more information.  I'm not sure if that is the same issue, but given than initially I see a TSL error, it probably is.

Comment 15 Daniel Walsh 2015-04-09 20:47:34 UTC

Well hopefully docker-io-1.5 will have the fix in the EPEL version.

Comment 16 Daniel Walsh 2015-04-14 19:59:55 UTC

Fixed in docker-1.6

Comment 17 John Morales 2015-06-17 11:56:35 UTC

I don't believe this is fixed, I'm seeing this still happening using docker-1.6:
Installed Packages
Name        : docker-io
Arch        : x86_64
Version     : 1.6.2
Release     : 1.el6
Size        : 21 M
Repo        : installed
From repo   : epel-testing

still seeing:
lr-x------ 1 root root 64 1434541872 25 -> /cgroup/memory/docker/.../memory.oom_control (deleted)
lrwx------ 1 root root 64 1434541872 26 -> [eventfd]
lr-x------ 1 root root 64 1434542118 27 -> /cgroup/memory/docker/.../memory.oom_control (deleted)
lrwx------ 1 root root 64 1434542118 28 -> [eventfd]

Comment 18 Daniel Walsh 2015-06-17 20:37:27 UTC

If there is a leak this is a separate new problem.  Please open a new bugzilla.

Comment 19 Lokesh Mandvekar 2016-04-02 06:37:02 UTC

Closing as docker-io on epel6 is a dead-end..

Note You need to log in before you can comment on or make changes to this bug.

adimania
admiller
arozansk
briemers
dwalsh
golang-updates
hushan.jia
jchaloup
jeder
jmario
jmorales
jperrin
lsm5
mattdm
mgoldman
miminar
srao
s
thrcka
tstclair
vbatts
vgoyal