Bug 540330

Summary: libvirtd causes oom - possible memory leak
Product: [Fedora] Fedora Reporter: Adrian A. Sender <asender>
Component: libvirtAssignee: Daniel Veillard <veillard>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: low    
Version: 12CC: berrange, clalance, crobinso, gianluca.cecchi, itamar, jforbes, rbinkhor, sven, tak, thomas.sjolshagen, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-05-10 16:16:23 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On: 540472, 540521    
Bug Blocks:    
Attachments:
Description Flags
virt-manager.log
none
virt-manager logs archive
none
Output from virt-manager --no-fork run
none
Python program to try & trigger a leak
none
Output from leak.py
none
valgrind --leak-check=full /usr/sbin/libvirtd output
none
valgrind log from overnight run with libvirt. none

Description Adrian A. Sender 2009-11-22 22:23:04 EST
This is happening on both my servers; eventually causes OOM killer to kill processes. 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2106 root      20   0 9943m 6.6g 2848 S  0.0 83.9   1:49.44 libvirtd   

0000000000400000   2220K r-x--  /usr/bin/qemu-kvm
000000000082b000     32K rw---  /usr/bin/qemu-kvm
0000000000833000   4144K rw---    [ anon ]
000000000196b000  10776K rw---    [ anon ]
0000003448000000    136K r-x--  /lib64/libncurses.so.5.7
0000003448022000   2044K -----  /lib64/libncurses.so.5.7
0000003448221000      4K rw---  /lib64/libncurses.so.5.7
0000003448800000    116K r-x--  /lib64/libtinfo.so.5.7
000000344881d000   2048K -----  /lib64/libtinfo.so.5.7
0000003448a1d000     16K rw---  /lib64/libtinfo.so.5.7
00000039c6600000   1468K r-x--  /usr/lib64/libcrypto.so.1.0.0
00000039c676f000   2044K -----  /usr/lib64/libcrypto.so.1.0.0
00000039c696e000    136K rw---  /usr/lib64/libcrypto.so.1.0.0
00000039c6990000     16K rw---    [ anon ]
00000039f4600000    120K r-x--  /lib64/ld-2.11.so
00000039f481d000      4K r----  /lib64/ld-2.11.so
00000039f481e000      4K rw---  /lib64/ld-2.11.so
00000039f481f000      4K rw---    [ anon ]
00000039f4a00000   1468K r-x--  /lib64/libc-2.11.so
00000039f4b6f000   2044K -----  /lib64/libc-2.11.so
00000039f4d6e000     16K r----  /lib64/libc-2.11.so
00000039f4d72000      4K rw---  /lib64/libc-2.11.so
00000039f4d73000     20K rw---    [ anon ]
00000039f4e00000      8K r-x--  /lib64/libdl-2.11.so
00000039f4e02000   2048K -----  /lib64/libdl-2.11.so
00000039f5002000      4K r----  /lib64/libdl-2.11.so
00000039f5003000      4K rw---  /lib64/libdl-2.11.so
00000039f5200000     92K r-x--  /lib64/libpthread-2.11.so
00000039f5217000   2044K -----  /lib64/libpthread-2.11.so
00000039f5416000      4K r----  /lib64/libpthread-2.11.so
00000039f5417000      4K rw---  /lib64/libpthread-2.11.so
00000039f5418000     16K rw---    [ anon ]
00000039f5600000    524K r-x--  /lib64/libm-2.11.so
00000039f5683000   2044K -----  /lib64/libm-2.11.so
00000039f5882000      4K r----  /lib64/libm-2.11.so
00000039f5883000      4K rw---  /lib64/libm-2.11.so
00000039f5a00000     28K r-x--  /lib64/librt-2.11.so
00000039f5a07000   2044K -----  /lib64/librt-2.11.so
00000039f5c06000      4K r----  /lib64/librt-2.11.so
00000039f5c07000      4K rw---  /lib64/librt-2.11.so
00000039f5e00000     84K r-x--  /lib64/libz.so.1.2.3
00000039f5e15000   2044K -----  /lib64/libz.so.1.2.3
00000039f6014000      4K rw---  /lib64/libz.so.1.2.3
00000039f6200000     44K r-x--  /usr/lib64/libpci.so.3.1.4
00000039f620b000   2048K -----  /usr/lib64/libpci.so.3.1.4
00000039f640b000      4K rw---  /usr/lib64/libpci.so.3.1.4
00000039f6600000     12K r-x--  /usr/lib64/libpulse-simple.so.0.0.3
00000039f6603000   2048K -----  /usr/lib64/libpulse-simple.so.0.0.3
00000039f6803000      4K rw---  /usr/lib64/libpulse-simple.so.0.0.3
00000039f6a00000     84K r-x--  /lib64/libresolv-2.11.so
00000039f6a15000   2048K -----  /lib64/libresolv-2.11.so
00000039f6c15000      4K r----  /lib64/libresolv-2.11.so
00000039f6c16000      4K rw---  /lib64/libresolv-2.11.so
00000039f6c17000      8K rw---    [ anon ]
00000039f6e00000     16K r-x--  /lib64/libcap-ng.so.0.0.0
00000039f6e04000   2044K -----  /lib64/libcap-ng.so.0.0.0
00000039f7003000      4K r----  /lib64/libcap-ng.so.0.0.0
00000039f7004000      4K rw---  /lib64/libcap-ng.so.0.0.0
00000039f7200000    252K r-x--  /lib64/libdbus-1.so.3.4.0
00000039f723f000   2048K -----  /lib64/libdbus-1.so.3.4.0
00000039f743f000      4K r----  /lib64/libdbus-1.so.3.4.0
00000039f7440000      4K rw---  /lib64/libdbus-1.so.3.4.0
00000039f8200000      8K r-x--  /usr/lib64/libXau.so.6.0.0
00000039f8202000   2048K -----  /usr/lib64/libXau.so.6.0.0
00000039f8402000      4K rw---  /usr/lib64/libXau.so.6.0.0
00000039f8a00000   1252K r-x--  /usr/lib64/libX11.so.6.3.0
00000039f8b39000   2048K -----  /usr/lib64/libX11.so.6.3.0
00000039f8d39000     24K rw---  /usr/lib64/libX11.so.6.3.0
00000039f8e00000    108K r-x--  /usr/lib64/libxcb.so.1.1.0
00000039f8e1b000   2044K -----  /usr/lib64/libxcb.so.1.1.0
00000039f901a000      4K rw---  /usr/lib64/libxcb.so.1.1.0
00000039f9e00000     68K r-x--  /usr/lib64/libXext.so.6.4.0
00000039f9e11000   2048K -----  /usr/lib64/libXext.so.6.4.0
00000039fa011000      4K rw---  /usr/lib64/libXext.so.6.4.0
00000039faa00000     60K r-x--  /usr/lib64/libXi.so.6.1.0
00000039faa0f000   2044K -----  /usr/lib64/libXi.so.6.1.0
00000039fac0e000      4K rw---  /usr/lib64/libXi.so.6.1.0
00000039ff200000    904K r-x--  /lib64/libasound.so.2.0.0
00000039ff2e2000   2048K -----  /lib64/libasound.so.2.0.0
00000039ff4e2000     28K rw---  /lib64/libasound.so.2.0.0
00000039ff4e9000      4K rw---    [ anon ]
00000039ffe00000     16K r-x--  /lib64/libuuid.so.1.3.0
00000039ffe04000   2044K -----  /lib64/libuuid.so.1.3.0
0000003a00003000      4K rw---  /lib64/libuuid.so.1.3.0
0000003a00200000    348K r-x--  /usr/lib64/libsndfile.so.1.0.20
0000003a00257000   2044K -----  /usr/lib64/libsndfile.so.1.0.20
0000003a00456000     12K rw---  /usr/lib64/libsndfile.so.1.0.20
0000003a00459000     16K rw---    [ anon ]
0000003a00600000    252K r-x--  /usr/lib64/libpulse.so.0.12.0
0000003a0063f000   2044K -----  /usr/lib64/libpulse.so.0.12.0
0000003a0083e000      8K rw---  /usr/lib64/libpulse.so.0.12.0
0000003a00e00000    308K r-x--  /usr/lib64/libpulsecommon-0.9.19.so
0000003a00e4d000   2044K -----  /usr/lib64/libpulsecommon-0.9.19.so
0000003a0104c000      8K rw---  /usr/lib64/libpulsecommon-0.9.19.so
0000003a01200000     32K r-x--  /lib64/libwrap.so.0.7.6
0000003a01208000   2048K -----  /lib64/libwrap.so.0.7.6
0000003a01408000      4K rw---  /lib64/libwrap.so.0.7.6
0000003a01409000      4K rw---    [ anon ]
0000003a02200000     92K r-x--  /usr/lib64/libICE.so.6.3.0
0000003a02217000   2048K -----  /usr/lib64/libICE.so.6.3.0
0000003a02417000      4K rw---  /usr/lib64/libICE.so.6.3.0
0000003a02418000     16K rw---    [ anon ]
0000003a02600000     28K r-x--  /usr/lib64/libSM.so.6.0.0
0000003a02607000   2048K -----  /usr/lib64/libSM.so.6.0.0
0000003a02807000      4K rw---  /usr/lib64/libSM.so.6.0.0
0000003a02a00000      8K r-x--  /lib64/libutil-2.11.so
0000003a02a02000   2044K -----  /lib64/libutil-2.11.so
0000003a02c01000      4K r----  /lib64/libutil-2.11.so
0000003a02c02000      4K rw---  /lib64/libutil-2.11.so
0000003a02e00000     12K r-x--  /lib64/libgpg-error.so.0.4.0
0000003a02e03000   2044K -----  /lib64/libgpg-error.so.0.4.0
0000003a03002000      4K rw---  /lib64/libgpg-error.so.0.4.0
0000003a03600000    452K r-x--  /lib64/libgcrypt.so.11.5.2
0000003a03671000   2048K -----  /lib64/libgcrypt.so.11.5.2
0000003a03871000     12K rw---  /lib64/libgcrypt.so.11.5.2
0000003a03874000      4K rw---    [ anon ]
0000003a04600000     20K r-x--  /usr/lib64/libXtst.so.6.1.0
0000003a04605000   2048K -----  /usr/lib64/libXtst.so.6.1.0
0000003a04805000      4K rw---  /usr/lib64/libXtst.so.6.1.0
0000003a04a00000     20K r-x--  /usr/lib64/libasyncns.so.0.3.1
0000003a04a05000   2044K -----  /usr/lib64/libasyncns.so.0.3.1
0000003a04c04000      4K rw---  /usr/lib64/libasyncns.so.0.3.1
0000003a06a00000     88K r-x--  /lib64/libnsl-2.11.so
0000003a06a16000   2044K -----  /lib64/libnsl-2.11.so
0000003a06c15000      4K r----  /lib64/libnsl-2.11.so
0000003a06c16000      4K rw---  /lib64/libnsl-2.11.so
0000003a06c17000      8K rw---    [ anon ]
0000003a06e00000    616K r-x--  /usr/lib64/libgnutls.so.26.14.12
0000003a06e9a000   2044K -----  /usr/lib64/libgnutls.so.26.14.12
0000003a07099000     28K rw---  /usr/lib64/libgnutls.so.26.14.12
0000003a07600000     64K r-x--  /usr/lib64/libtasn1.so.3.1.6
0000003a07610000   2048K -----  /usr/lib64/libtasn1.so.3.1.6
0000003a07810000      4K rw---  /usr/lib64/libtasn1.so.3.1.6
0000003a08200000    432K r-x--  /usr/lib64/libSDL-1.2.so.0.11.2
0000003a0826c000   2048K -----  /usr/lib64/libSDL-1.2.so.0.11.2
0000003a0846c000     12K rw---  /usr/lib64/libSDL-1.2.so.0.11.2
0000003a0846f000    192K rw---    [ anon ]
0000003a0c200000   1468K r-x--  /lib64/libdb-4.7.so
0000003a0c36f000   2044K -----  /lib64/libdb-4.7.so
0000003a0c56e000     24K rw---  /lib64/libdb-4.7.so
00007fa040000000    516K rw---    [ anon ]
00007fa040081000  65020K -----    [ anon ]
00007fa047ffe000      4K rw---    [ anon ]
00007fa047fff000 2097152K rw---    [ anon ]
00007fa0c7fff000      4K rw---    [ anon ]
00007fa0c8000000    208K rw---    [ anon ]
00007fa0c8034000  65328K -----    [ anon ]
00007fa0cd9db000      4K -----    [ anon ]
00007fa0cd9dc000  10240K rw---    [ anon ]
00007fa0ce3dc000     20K r-x--  /usr/lib64/sasl2/libsasldb.so.2.0.23
00007fa0ce3e1000   2044K -----  /usr/lib64/sasl2/libsasldb.so.2.0.23
00007fa0ce5e0000      4K rw---  /usr/lib64/sasl2/libsasldb.so.2.0.23
00007fa0ce5e1000     16K r-x--  /usr/lib64/sasl2/libanonymous.so.2.0.23
00007fa0ce5e5000   2044K -----  /usr/lib64/sasl2/libanonymous.so.2.0.23
00007fa0ce7e4000      4K rw---  /usr/lib64/sasl2/libanonymous.so.2.0.23
00007fa0ce7e5000     48K r-x--  /usr/lib64/sasl2/libdigestmd5.so.2.0.23
00007fa0ce7f1000   2044K -----  /usr/lib64/sasl2/libdigestmd5.so.2.0.23
00007fa0ce9f0000      4K rw---  /usr/lib64/sasl2/libdigestmd5.so.2.0.23
00007fa0ce9f1000     16K r-x--  /usr/lib64/sasl2/liblogin.so.2.0.23
00007fa0ce9f5000   2044K -----  /usr/lib64/sasl2/liblogin.so.2.0.23
00007fa0cebf4000      4K rw---  /usr/lib64/sasl2/liblogin.so.2.0.23
00007fa0cebf5000     16K r-x--  /usr/lib64/sasl2/libcrammd5.so.2.0.23
00007fa0cebf9000   2048K -----  /usr/lib64/sasl2/libcrammd5.so.2.0.23
00007fa0cedf9000      4K rw---  /usr/lib64/sasl2/libcrammd5.so.2.0.23
00007fa0cedfa000     16K r-x--  /usr/lib64/sasl2/libplain.so.2.0.23
00007fa0cedfe000   2044K -----  /usr/lib64/sasl2/libplain.so.2.0.23
00007fa0ceffd000      4K rw---  /usr/lib64/sasl2/libplain.so.2.0.23
00007fa0ceffe000      4K rw---    [ anon ]
00007fa0cefff000  16384K rw---    [ anon ]
00007fa0cffff000      4K rw---    [ anon ]
00007fa0d0000000    188K rw---    [ anon ]
00007fa0d002f000  65348K -----    [ anon ]
00007fa0d40ef000   2108K rw---    [ anon ]
00007fa0d42fe000    520K rw---    [ anon ]
00007fa0d4380000      4K rw---    [ anon ]
00007fa0d4381000    128K rw---    [ anon ]
00007fa0d43a1000      8K rw---    [ anon ]
00007fa0d43a3000    128K rw---    [ anon ]
00007fa0d43c3000      4K rw---    [ anon ]
00007fa0d43e2000    396K rw---    [ anon ]
00007fa0d4445000      4K -----    [ anon ]
00007fa0d4446000  10240K rw---    [ anon ]
00007fa0d4e46000      4K -----    [ anon ]
00007fa0d4e47000  10240K rw---    [ anon ]
00007fa0d5847000     48K r-x--  /lib64/libnss_files-2.11.so
00007fa0d5853000   2044K -----  /lib64/libnss_files-2.11.so
00007fa0d5a52000      4K r----  /lib64/libnss_files-2.11.so
00007fa0d5a53000      4K rw---  /lib64/libnss_files-2.11.so
00007fa0d5a64000     20K rw---    [ anon ]
00007fa0d5a69000    356K r-x--  /usr/lib64/libfreebl3.so
00007fa0d5ac2000   2048K -----  /usr/lib64/libfreebl3.so
00007fa0d5cc2000      4K rw---  /usr/lib64/libfreebl3.so
00007fa0d5cc3000     32K rw---    [ anon ]
00007fa0d5ccb000     28K r-x--  /lib64/libcrypt-2.11.so
00007fa0d5cd2000   2048K -----  /lib64/libcrypt-2.11.so
00007fa0d5ed2000      4K r----  /lib64/libcrypt-2.11.so
00007fa0d5ed3000      4K rw---  /lib64/libcrypt-2.11.so
00007fa0d5ed4000    212K rw---    [ anon ]
00007fa0d5f09000    100K r-x--  /usr/lib64/libsasl2.so.2.0.23
00007fa0d5f22000   2048K -----  /usr/lib64/libsasl2.so.2.0.23
00007fa0d6122000      4K rw---  /usr/lib64/libsasl2.so.2.0.23
00007fa0d6123000      8K rw---    [ anon ]
00007fa0d612b000     12K rw-s-    [ anon ]
00007fa0d612e000     12K rw-s-    [ anon ]
00007fa0d6131000     20K rw---    [ anon ]
00007fff88356000     84K rw---    [ stack ]
00007fff883eb000      4K r-x--    [ anon ]
ffffffffff600000      4K r-x--    [ anon ]
 total          2470520K
Comment 1 Adrian A. Sender 2009-11-22 23:47:53 EST
Using strace & ps I've gathered further diagnostic info on this process's behaviour:

[root@core-01 ~]# strace -p 13931

Process 13931 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 1
accept(13, {sa_family=AF_FILE, NULL}, [2]) = 16
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = -1 EOPNOTSUPP (Operation not supported)
fcntl(16, F_GETFD)                      = 0
fcntl(16, F_SETFD, FD_CLOEXEC)          = 0
fcntl(16, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
brk(0x2735000)                          = 0x2735000
getsockopt(16, SOL_SOCKET, SO_PEERCRED, "\362@\0\0\0\0\0\0\0\0\0\0", [12]) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
brk(0x2775000)                          = 0x2775000
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0B\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\1"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1"..., 56) = 56
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0n\0\0\0\0\0\0\0\2\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0L \0\200\206\0\0\0\1\0\0\0n\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\16"..., 76) = 76
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\3\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0\3\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\4"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\4\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0\4\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\5\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\2\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0"..., 56) = 56
close(16)                               = 0
brk(0x26f5000)                          = 0x26f5000
close(18)                               = 0
close(17)                               = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}], 10, -1) = 1 ([{fd=13, revents=POLLIN}])
accept(13, {sa_family=AF_FILE, NULL}, [2]) = 16
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = -1 EOPNOTSUPP (Operation not supported)
fcntl(16, F_GETFD)                      = 0
fcntl(16, F_SETFD, FD_CLOEXEC)          = 0
fcntl(16, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
brk(0x2735000)                          = 0x2735000
getsockopt(16, SOL_SOCKET, SO_PEERCRED, "\3A\0\0\0\0\0\0\0\0\0\0", [12]) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
brk(0x2775000)                          = 0x2775000
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0B\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\1"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1"..., 56) = 56
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0n\0\0\0\0\0\0\0\2\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0L \0\200\206\0\0\0\1\0\0\0n\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\16"..., 76) = 76
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\3\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x681d40, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x681d40, FUTEX_WAKE_PRIVATE, 1)  = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0\3\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\4"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\4\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0\4\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\5\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\2\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0"..., 56) = 56
close(16)                               = 0
brk(0x26f5000)                          = 0x26f5000
close(18)                               = 0
close(17)                               = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}], 10, -1) = 1 ([{fd=13, revents=POLLIN}])
accept(13, {sa_family=AF_FILE, NULL}, [2]) = 16
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = -1 EOPNOTSUPP (Operation not supported)
fcntl(16, F_GETFD)                      = 0
fcntl(16, F_SETFD, FD_CLOEXEC)          = 0
fcntl(16, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
brk(0x2735000)                          = 0x2735000
getsockopt(16, SOL_SOCKET, SO_PEERCRED, "\7A\0\0\0\0\0\0\0\0\0\0", [12]) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
brk(0x2775000)                          = 0x2775000
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0B\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\1"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1"..., 56) = 56
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0n\0\0\0\0\0\0\0\2\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x681d40, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x681d40, FUTEX_WAKE_PRIVATE, 1)  = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0L \0\200\206\0\0\0\1\0\0\0n\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\16"..., 76) = 76
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0,", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\27\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\vblue"..., 40) = 40
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x25ee440, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0\\ \0\200\206\0\0\0\1\0\0\0\27\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\v"..., 92) = 92
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0@", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\20\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\vblue"..., 60) = 60
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f868c0dc600, FUTEX_WAKE_PRIVATE, 1) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0X \0\200\206\0\0\0\1\0\0\0\20\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\1"..., 88) = 88
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\5\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\2\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0"..., 56) = 56
close(16)                               = 0
brk(0x26f5000)                          = 0x26f5000
close(18)                               = 0
close(17)                               = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}], 10, -1) = 1 ([{fd=13, revents=POLLIN}])
accept(13, {sa_family=AF_FILE, NULL}, [2]) = 16
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = -1 EOPNOTSUPP (Operation not supported)
fcntl(16, F_GETFD)                      = 0
fcntl(16, F_SETFD, FD_CLOEXEC)          = 0
fcntl(16, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
brk(0x2735000)                          = 0x2735000
getsockopt(16, SOL_SOCKET, SO_PEERCRED, "\nA\0\0\0\0\0\0\0\0\0\0", [12]) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
brk(0x2775000)                          = 0x2775000
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0B\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\1"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1"..., 56) = 56
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0n\0\0\0\0\0\0\0\2\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0L \0\200\206\0\0\0\1\0\0\0n\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\16"..., 76) = 76
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\27\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\10rhel"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f868c0dc600, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x681d40, FUTEX_WAKE_PRIVATE, 1)  = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0X \0\200\206\0\0\0\1\0\0\0\27\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\10"..., 88) = 88
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0<", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\20\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\10rhel"..., 56) = 56
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0X \0\200\206\0\0\0\1\0\0\0\20\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\1"..., 88) = 88
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\5\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\2\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0"..., 56) = 56
close(16)                               = 0
brk(0x26f5000)                          = 0x26f5000
close(18)                               = 0
close(17)                               = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}], 10, -1) = 1 ([{fd=13, revents=POLLIN}])
accept(13, {sa_family=AF_FILE, NULL}, [2]) = 16
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = -1 EOPNOTSUPP (Operation not supported)
fcntl(16, F_GETFD)                      = 0
fcntl(16, F_SETFD, FD_CLOEXEC)          = 0
fcntl(16, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
brk(0x2735000)                          = 0x2735000
getsockopt(16, SOL_SOCKET, SO_PEERCRED, "\26A\0\0\0\0\0\0\0\0\0\0", [12]) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
brk(0x2775000)                          = 0x2775000
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x681d40, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x681d40, FUTEX_WAKE_PRIVATE, 1)  = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0B\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\1"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1"..., 56) = 56
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0n\0\0\0\0\0\0\0\2\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x681d40, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x681d40, FUTEX_WAKE_PRIVATE, 1)  = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0L \0\200\206\0\0\0\1\0\0\0n\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\16"..., 76) = 76
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\3\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0\3\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\4"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\4\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0\4\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\0"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\5\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\2\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0"..., 56) = 56
close(16)                               = 0
brk(0x26f5000)                          = 0x26f5000
close(18)                               = 0
close(17)                               = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}], 10, -1) = 1 ([{fd=13, revents=POLLIN}])
accept(13, {sa_family=AF_FILE, NULL}, [2]) = 16
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = -1 EOPNOTSUPP (Operation not supported)
fcntl(16, F_GETFD)                      = 0
fcntl(16, F_SETFD, FD_CLOEXEC)          = 0
fcntl(16, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
brk(0x2735000)                          = 0x2735000
getsockopt(16, SOL_SOCKET, SO_PEERCRED, "\34A\0\0\0\0\0\0\0\0\0\0", [12]) = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
brk(0x2775000)                          = 0x2775000
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x681d40, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x681d40, FUTEX_WAKE_PRIVATE, 1)  = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 2 ([{fd=5, revents=POLLIN}, {fd=16, revents=POLLOUT}])
read(5, "\0", 1)                        = 1
write(16, "\0\0\0@ \0\200\206\0\0\0\1\0\0\0B\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\1"..., 64) = 64
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\1\0\0\0\0"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x26d4520, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x25ee440, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\1"..., 56) = 56
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0n\0\0\0\0\0\0\0\2\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0L \0\200\206\0\0\0\1\0\0\0n\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\16"..., 76) = 76
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0(", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\27\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\10winx"..., 36) = 36
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0X \0\200\206\0\0\0\1\0\0\0\27\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\10"..., 88) = 88
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0<", 4)                  = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\20\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\10winx"..., 56) = 56
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x26d4520, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0X \0\200\206\0\0\0\1\0\0\0\20\0\0\0\1\0\0\0\4\0\0\0\0\0\0\0\1"..., 88) = 88
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=16, revents=POLLIN}])
read(16, "\0\0\0\34", 4)                = 4
read(16, " \0\200\206\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\5\0\0\0\0", 24) = 24
futex(0x25ee46c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x25ee468, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN}], 11, -1) = 1 ([{fd=5, revents=POLLIN}])
read(5, "\0", 1)                        = 1
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}, {fd=16, events=POLLIN|POLLOUT}], 11, -1) = 1 ([{fd=16, revents=POLLOUT}])
write(16, "\0\0\0008 \0\200\206\0\0\0\1\0\0\0\2\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0"..., 56) = 56
close(16)                               = 0
brk(0x26f5000)                          = 0x26f5000
close(18)                               = 0
close(17)                               = 0
poll([{fd=5, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, events=POLLERR|POLLHUP}, {fd=10, events=POLLERR|POLLHUP}, {fd=11, events=POLLERR|POLLHUP}, {fd=12, events=POLLIN}, {fd=3, events=POLLIN}, {fd=13, events=POLLIN|POLLERR|POLLHUP}, {fd=14, events=POLLIN|POLLERR|POLLHUP}, {fd=15, events=POLLIN}], 10, -1^C <unfinished ...>

At the end of every one of this poll interval, the rss of this process is consistently growing:

[root@core-01 ~]# while true; do ps -eo rss,pid,cmd|egrep libvirtd|egrep -v egrep; sleep 30; done
11236 13931 libvirtd --daemon
11240 13931 libvirtd --daemon
11240 13931 libvirtd --daemon
11244 13931 libvirtd --daemon
11276 13931 libvirtd --daemon
11276 13931 libvirtd --daemon
11276 13931 libvirtd --daemon
11276 13931 libvirtd --daemon
11280 13931 libvirtd --daemon
11280 13931 libvirtd --daemon
11280 13931 libvirtd --daemon
11280 13931 libvirtd --daemon
11284 13931 libvirtd --daemon
Comment 2 Adrian A. Sender 2009-11-23 00:04:20 EST
I have also tested this on my laptop; and noticed everytime I open and close virtmanager rss memory doubles

 7048  9758 libvirtd --daemon  -started libvirt
 24648  9758 libvirtd --daemon -opened virt-manager; closed virt-manager
 41920  9758 libvirtd --daemon -opened virt-manager; closed virt-manager
 44272  9758 libvirtd --daemon -opened virt-manager; closed virt-manager
 64916  9758 libvirtd --daemon -opened virt-manager
Comment 3 Adrian A. Sender 2009-11-23 00:09:57 EST
Colleague tested on Fedora 11 - same problem exists- but on a lower scale rather than 20M per iteration on F12, 1M on F11; how to reproduce... open and close libvirt 100 times and your machine will crash!
Comment 4 Daniel Veillard 2009-11-23 04:43:19 EST
What version of libvirt are you running ? i.e. output of 
rpm -q libvirt

Daniel
Comment 5 Adrian A. Sender 2009-11-23 04:57:42 EST
libvirt-0.7.1-15.fc12.x86_64 shall I try rawhide, as I said this appears on Fedora 11 as well but not as severe (or in test case less virtual machine) - you should be able to reproduce quite easily.
Comment 6 Daniel Veillard 2009-11-23 07:52:38 EST
I was able to reproduce the problem from comment #3 on an F12 machine.
Then I updated to a local build of libvirt-0.7.4 , restarted libvirtd
service and ran the same experiment, and the leak seems gone.

http://libvirt.org/git/?p=libvirt.git&a=search&h=e7c78b0a94a2c67ff7947a205100ed544bd8fab3&st=commit&s=leak

there have been a number of memory leak fixes in the last few weeks,
this probably explains that, so yes I would suggest to update to a rebuild
of 0.7.4, and tell us if you still see the issue,

  thanks,

Daniel
Comment 7 Daniel Berrange 2009-11-23 08:02:18 EST
I can still reproduce the problem on latest GIT trees, so I don't think 0.7.4  fixes it (at least not entirely - there's always the possibility of multiple bugs )
Comment 8 Daniel Berrange 2009-11-23 08:15:07 EST
I've narrowed this down to being a problem relating to the interface drivers, since you turn virRegisterInterfaceDriver() into a no-op all the leaks go away.
Comment 9 Daniel Berrange 2009-11-23 08:50:44 EST
I have found & fixed bug in netcf, but it turns out this only impacts libvirtd when run as non-root.

https://bugzilla.redhat.com/show_bug.cgi?id=540472

So my comment #8 is probably incorrect wrt your scenario of running libvirtd as root.
Comment 10 Daniel Berrange 2009-11-23 08:56:35 EST
Can you provide the $HOME/.virt-manager/virt-manager.log file, and also the /var/log/messages
Comment 11 Adrian A. Sender 2009-11-23 09:21:04 EST
There are two independent bugs here one with open and closing virt-manager; the other for polling the virtual machines. (refer to comment 1 and comment 3).

After running libvirt for ~7 hours you can see the effects from comment 1 taking place.

I dont see anything in virt-manager.log or messages to be useful, this is happening on two independent machines; as well as laptop. Eventually OOM is called and start killing off random processes.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
13931 root      20   0 4749m 4.3g 3756 S  0.0 55.5   0:54.81 libvirtd           
 3710 qemu      20   0  721m 472m 1284 S  7.1  5.9 121:48.89 qemu-kvm           
 3462 qemu      20   0 2412m 435m 1272 S  5.3  5.4 157:13.18 qemu-kvm           
19904 qemu      20   0  795m 391m 1280 S  4.6  4.9  91:48.67 qemu-kvm
Comment 12 Adrian A. Sender 2009-11-23 09:34:34 EST
One additional observation from the original description; can you check rss from libvirtd and compare it to pmap total.
Comment 13 Daniel Berrange 2009-11-23 10:47:47 EST
Please provide the virt-manager.log file, because I want to see how virt-manager is connecting.
Comment 14 Adrian A. Sender 2009-11-23 17:13:53 EST
Created attachment 373258 [details]
virt-manager.log
Comment 15 Anders 2009-11-30 03:30:30 EST
I can confirm seeing this same problem as per comment#1. Four VM's running, three idle, one active (upgrading F11 -> F12). Left overnight (about 8 hours), libvirt was taking up 40% RAM (of 8GB) and the system was all the way out in swap.
Recoverable, when turning off all the VM's and restarting libvirtd service, but means I can't work with VM's like I could on F11.

$ rpm -q libvirt
libvirt-0.7.1-15.fc12.x86_64

Attaching the virt-manager logs as well. (This system was updated to F12, from F11, on the 29th, early afternoon. You can see the gap in the virt-manager logs between 14:09 and 18:13.)

I can do some testing, so if there is a updates-testing or koji build that can be installed on F12, I'm game to run it.

/Anders
Comment 16 Anders 2009-11-30 03:32:00 EST
Created attachment 374703 [details]
virt-manager logs archive

Made an archive as the problem spans the two logs.
Comment 17 Daniel Berrange 2009-11-30 04:43:48 EST
> Recoverable, when turning off all the VM's and restarting libvirtd service, but
> means I can't work with VM's like I could on F11.

FYI, you can safely restart  libvirtd while VMs are running - i'll leave them running untouched and redetect them when it starts again.
Comment 18 Adrian A. Sender 2009-11-30 04:55:13 EST
Hi Daniel,

I can restart libvirtd and the vm's wont be effected, however when other services depend on libvirtd ie - rgmanager its a different story and causes all sorts of problems.
Comment 19 Daniel Berrange 2009-11-30 06:05:55 EST
I'm still trying to figure out what's different about your environment, since I don't see any obvious leaks on my F12 system, apart from the netcf things I mentioned which can't account for many GB of leaking. 

Can you show me how your virt-manager is configured, by attaching output of:

  # gconftool-2 -R /apps/virt-manager


And, then get a trace of what libivrt APIs it is running over the space of about 20 seconds by running


  # LIBVIRT_LOG_FILTERS="1:libvirt" LIBVIRT_LOG_OUTPUTS="1:stderr" virt-manager --no-fork
Comment 20 Thomas Sjolshagen 2009-11-30 07:49:54 EST
I, too, am seeing the leaks on Fedora 12. Tried libvirt 0.7.1. and 0.7.4 with the same result. In under 8 hours, libvirt will consume near 5GB of RAM. Only happens while there are guests running on the system (I've got 8) and happens faster if I leave virt-manager running.

Am using rgmanager to start/stop/migrate my guests so I suspect it's rather frequent checks of the guest state is aggravating the issue further.

Just for "fun" I ran the above mentioned virt-manager --no-fork and dumped the contents along with ps auxx output into the attached log file. This system is presently using libvirt 0.7.1.
Comment 21 Thomas Sjolshagen 2009-11-30 07:50:46 EST
Created attachment 374740 [details]
Output from virt-manager --no-fork run
Comment 22 Daniel Berrange 2009-11-30 08:47:02 EST
Well that virt-manager log identified one difference with my setup - my virt-manager was not querying disk or network device statistics, but even enabling that doesn't trigger any memory leak.

I'm attaching a short python program - I'd appreciate info on whether this python program causes libvirtd to leak - make sure no other apps like virt-manager / rgmanager  are connected to libvirt while running this - i want this demo program being the only app running with libvirt. Make sure at least a couple of guests are running too.
Comment 23 Daniel Berrange 2009-11-30 08:48:04 EST
Created attachment 374752 [details]
Python program to try & trigger a leak

Simple run as root

 # python leak.py
Comment 24 Thomas Sjolshagen 2009-11-30 09:48:29 EST
Had 3 guests running and ran the leak.py script. Output being attached.

Looped some ps output at the same time in a different terminal and did not see a leak.
Comment 25 Thomas Sjolshagen 2009-11-30 09:49:59 EST
Created attachment 374766 [details]
Output from leak.py
Comment 26 Thomas Sjolshagen 2009-11-30 10:34:09 EST
# gconftool-2 -R /apps/virt-manager
 system-tray = false
 /apps/virt-manager/urls:
  kickstart = []
  url-list-length = 10
  media = [http://astromirror.uchicago.edu/fedora/linux/releases/9/Fedora/x86_64/os/,http://download.fedora.redhat.com/pub/fedora/linux/releases/8/Fedora/x86_64/os/,http://download.fedora.redhat.com/pub/fedora/linux/releases/8/Fedora/i386/os/,http://mirrors.kernel.org/fedora/releases/9/Fedora/x86_64/,http://centos.omnispring.com/5.1/os/x86_64/,http://centos.omnispring.com/5.1/os/x86_64/isolinux/,http://centos.omnispring.com/5.1/os/x86_64,http://centos.omnispring.com/,http://centos.take2hosting.com/5.1/os/x86_64/images,http://centos.take2hosting.com/5.1/os/x86_64/]
 /apps/virt-manager/vmlist-fields:
  cpu_usage = false
  domain_id = true
  memory_usage = false
  virtual_cpus = true
  stats-type = 0
 /apps/virt-manager/paths:
  default-screenshot-path = 
  default-save-path = 
  default-image-path = 
  default-restore-path = 
  default-media-path = 
 /apps/virt-manager/stats:
  history-length = 120
  enable-net-poll = true
  update-interval = 1
  enable-disk-poll = true
  enable-mem-poll = true
  enable-cpu-poll = true
 /apps/virt-manager/console:
  scaling = 1
  keygrab = 2
  grab-notify = false
  popup = 1
 /apps/virt-manager/new-vm:
  local-sound = true
  remote-sound = false
 /apps/virt-manager/connections:
  uris = [qemu+tls://virt0-backup/system,qemu:///system]
  autoconnect = []
 /apps/virt-manager/details:
  show-toolbar = false
Comment 27 Daniel Berrange 2009-11-30 10:58:24 EST
Opps, I didn't actually need the output from leak.py, I just wanted to know if libvirtd leaked while running it :-)

Anyway, since it didn't leak, at least that bit matches behaviour I see. So it does not appear to be related to the domain XML, Info, block stats, or interface stats functions. That doesn't really leave me any other ideas for what's causing the leak though.
Comment 28 Daniel Berrange 2009-11-30 11:44:05 EST
I think we need to try and get a valgrind trace of the libvirtd daemon. So if someone here is able to help....

Start a few VMs when libvirtd is running normally, then without stopping the VMs, as root, run:

 # debuginfo-install libvirt
 # service libvirtd stop
 # valgrind --leak-check=full /usr/sbin/libvirtd


and then bring up virt-manager and connect it to libvirtd, and let it run for a few minutes. Then Ctrl-C the valgrind process and attach the logs it prints out to this BZ.
Comment 29 Thomas Sjolshagen 2009-11-30 14:34:25 EST
Just put # virsh domstate <domainname> into a perpetual while loop for 6 domains:

# while :; do virsh domstate database ; virsh domstate imap ; virsh domstate imap1-cluster ; virsh domstate imap2-cluster ; virsh domstate mail ; virsh domstate www; done

Not that it matters, but the domains were all stopped.

Ran a separate ps auxx loop to for libvirtd's memory consumption:

# while : ; do ps auxww | grep libvirtd | grep -v grep ; sleep 2 ; done

And the output from the libvirtd while loop:

root      1158  0.0  0.1 540408 10400 ?        Sl   10:10   0:00 libvirtd --daemon
root      1158  0.0  0.1 540408 10400 ?        Sl   10:10   0:00 libvirtd --daemon
root      1158  0.0  0.1 540408 10400 ?        Sl   10:10   0:00 libvirtd --daemon
root      1158  0.0  0.1 540408 10400 ?        Sl   10:10   0:00 libvirtd --daemon
root      1158  0.0  0.1 540408 10400 ?        Sl   10:10   0:00 libvirtd --daemon
root      1158  0.0  0.1 540408 10400 ?        Sl   10:10   0:00 libvirtd --daemon
root      1158  0.0  0.1 540408 10404 ?        Sl   10:10   0:00 libvirtd --daemon
-- started # virsh domstate loop ------
root      1158  0.0  0.1 540920 10584 ?        Rl   10:10   0:01 libvirtd --daemon
root      1158  0.0  0.1 540408 10580 ?        Sl   10:10   0:02 libvirtd --daemon
root      1158  0.0  0.1 540920 10876 ?        Sl   10:10   0:03 libvirtd --daemon
root      1158  0.0  0.1 540920 10812 ?        Rl   10:10   0:03 libvirtd --daemon
root      1158  0.0  0.1 540408 10700 ?        Sl   10:10   0:04 libvirtd --daemon
root      1158  0.0  0.1 540408 10776 ?        Sl   10:10   0:05 libvirtd --daemon
root      1158  0.0  1.6 540920 133436 ?       Rl   10:10   0:06 libvirtd --daemon
root      1158  0.0  3.1 540920 259244 ?       Sl   10:10   0:07 libvirtd --daemon
root      1158  0.0  4.6 868604 384304 ?       Sl   10:10   0:08 libvirtd --daemon
root      1158  0.0  6.2 868088 509248 ?       Sl   10:10   0:09 libvirtd --daemon
root      1158  0.0  7.7 868088 634848 ?       Sl   10:10   0:10 libvirtd --daemon
root      1158  0.0  8.8 1195768 724556 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 828704 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 828704 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 828704 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 828704 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 829472 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon

  --- Terminated # virsh domstate loop ----

root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 830236 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 831004 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 831004 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 831004 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 831004 ?      Sl   10:10   0:11 libvirtd --daemon
root      1158  0.0 10.1 1195768 831004 ?      Sl   10:10   0:11 libvirtd --daemon
Comment 30 Thomas Sjolshagen 2009-11-30 14:51:50 EST
Created attachment 374865 [details]
valgrind --leak-check=full /usr/sbin/libvirtd output

# valgrind --leak-check=full /usr/sbin/libvirtd > ./valgrind-out.txt 2>&1

Then looped virsh domstate <dom1> ; virsh domstate <dom2> ; etc for a short while (20 seconds or so I'd guesstimate).
Comment 31 Gianluca Cecchi 2009-12-01 05:00:58 EST
I'm experimenting the same after updating system from F11 x86_64+rawvirt repo
to F12.
Could it be anything related to users previously running rawvirt on f11, or
does it happen also to fresh F12 environments?
My system is an HP ProLiant BL480c G1 with 12Gb of ram.
I have 4 guests, all x86_64: 
2 with CentOS 5.3 and 1Gb of ram each
2 with CentOS 5.4 and 3,5Gb of ram each.

boot of host on 26/11 about 19:20 with the first two guests configured to
auto-start
On 27/11 at 12:03 I start the other two guests in centos 5.4
The day after at 06:55

Nov 28 03:27:06 virtfed logrotate: ALERT exited abnormally with [1]
Nov 28 06:55:01 virtfed kernel: qemu-kvm invoked oom-killer:
gfp_mask=0x201da, order=0, oomkilladj=0
Nov 28 06:55:01 virtfed kernel: qemu-kvm cpuset=/ mems_allowed=0
Nov 28 06:55:01 virtfed kernel: Pid: 405, comm: qemu-kvm Tainted: G
    W  2.6.31.5-127.fc12.x86_64 #1
Nov 28 06:55:01 virtfed kernel: Call Trace:
Nov 28 06:55:01 virtfed kernel: [<ffffffff8108a7d7>] ?
cpuset_print_task_mems_allowed+0x91/0x9d
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c38e9>]
oom_kill_process+0x98/0x256
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3d6a>] ?
select_bad_process+0xa3/0x102
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3e53>] __out_of_memory+0x8a/0x99
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3fc5>] out_of_memory+0x163/0x195
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c75c1>]
__alloc_pages_nodemask+0x491/0x584
Nov 28 06:55:01 virtfed kernel: [<ffffffff810eb1c4>]
alloc_pages_current+0x95/0x9e
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c15c1>]
__page_cache_alloc+0x5f/0x61
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c9235>]
__do_page_cache_readahead+0x98/0x176
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c9334>] ra_submit+0x21/0x25
Nov 28 06:55:01 virtfed kernel: [<ffffffff810c1abd>] filemap_fault+0x193/0x317
Nov 28 06:55:01 virtfed kernel: [<ffffffff810d6641>] __do_fault+0x54/0x3c4
Nov 28 06:55:01 virtfed kernel: [<ffffffff81017bc1>] ? read_tsc+0x9/0x1b
Nov 28 06:55:01 virtfed kernel: [<ffffffff810d881a>]
handle_mm_fault+0x2f6/0x705
Nov 28 06:55:01 virtfed kernel: [<ffffffff8106a9ff>] ?
__hrtimer_start_range_ns+0x2bb/0x2cd
Nov 28 06:55:01 virtfed kernel: [<ffffffff8141cfe1>] do_page_fault+0x281/0x299
Nov 28 06:55:01 virtfed kernel: [<ffffffff8141af75>] page_fault+0x25/0x30
Nov 28 06:55:01 virtfed kernel: Mem-Info:
Nov 28 06:55:01 virtfed kernel: Node 0 DMA per-cpu:
Nov 28 06:55:01 virtfed kernel: CPU    0: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: CPU    1: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: CPU    2: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: CPU    3: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: CPU    4: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: CPU    5: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: CPU    6: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: CPU    7: hi:    0, btch:   1 usd:   0
Nov 28 06:55:01 virtfed kernel: Node 0 DMA32 per-cpu:
Nov 28 06:55:01 virtfed kernel: CPU    0: hi:  186, btch:  31 usd:  86
Nov 28 06:55:01 virtfed kernel: CPU    1: hi:  186, btch:  31 usd: 136
Nov 28 06:55:01 virtfed kernel: CPU    2: hi:  186, btch:  31 usd:  51
Nov 28 06:55:01 virtfed kernel: CPU    3: hi:  186, btch:  31 usd: 126
Nov 28 06:55:01 virtfed kernel: CPU    4: hi:  186, btch:  31 usd:  86
Nov 28 06:55:01 virtfed kernel: CPU    5: hi:  186, btch:  31 usd:  34
Nov 28 06:55:01 virtfed kernel: CPU    6: hi:  186, btch:  31 usd: 171
Nov 28 06:55:01 virtfed kernel: CPU    7: hi:  186, btch:  31 usd: 169
Nov 28 06:55:01 virtfed kernel: Node 0 Normal per-cpu:
Nov 28 06:55:01 virtfed kernel: CPU    0: hi:  186, btch:  31 usd: 173
Nov 28 06:55:01 virtfed kernel: CPU    1: hi:  186, btch:  31 usd:  42
Nov 28 06:55:01 virtfed kernel: CPU    2: hi:  186, btch:  31 usd: 160
Nov 28 06:55:01 virtfed kernel: CPU    3: hi:  186, btch:  31 usd: 109
Nov 28 06:55:01 virtfed kernel: CPU    4: hi:  186, btch:  31 usd: 112
Nov 28 06:55:01 virtfed kernel: CPU    5: hi:  186, btch:  31 usd: 112
Nov 28 06:55:01 virtfed kernel: CPU    6: hi:  186, btch:  31 usd: 174
Nov 28 06:55:01 virtfed kernel: CPU    7: hi:  186, btch:  31 usd: 118
Nov 28 06:55:01 virtfed kernel: Active_anon:2537459 active_file:3
inactive_anon:364350
Nov 28 06:55:01 virtfed kernel: inactive_file:160 unevictable:12562
dirty:0 writeback:0 unstable:0
Nov 28 06:55:01 virtfed kernel: free:16256 slab:29833 mapped:7872
pagetables:10697 bounce:0
Nov 28 06:55:01 virtfed kernel: Node 0 DMA free:15824kB min:16kB
low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB present:15320kB pages_scanned:0
all_unreclaimable? yes
Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 3254 12092 12092
Nov 28 06:55:01 virtfed kernel: Node 0 DMA32 free:39060kB min:3784kB
low:4728kB high:5676kB active_anon:2510452kB inactive_anon:502304kB
active_file:12kB inactive_file:244kB unevictable:520kB
present:3332660kB pages_scanned:120 all_unreclaimable? no
Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 0 8837 8837
Nov 28 06:55:01 virtfed kernel: Node 0 Normal free:10140kB min:10280kB
low:12848kB high:15420kB active_anon:7639384kB inactive_anon:955096kB
active_file:0kB inactive_file:396kB unevictable:49728kB
present:9049596kB pages_scanned:0 all_unreclaimable? no
Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 0 0 0
Nov 28 06:55:01 virtfed kernel: Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB
2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB
Nov 28 06:55:01 virtfed kernel: Node 0 DMA32: 3793*4kB 2168*8kB
68*16kB 11*32kB 4*64kB 6*128kB 3*256kB 2*512kB 2*1024kB 0*2048kB
0*4096kB = 38820kB
Nov 28 06:55:01 virtfed kernel: Node 0 Normal: 1948*4kB 5*8kB 0*16kB
0*32kB 2*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB =
10264kB
Nov 28 06:55:01 virtfed kernel: 26703 total pagecache pages
Nov 28 06:55:01 virtfed kernel: 18747 pages in swap cache
Nov 28 06:55:01 virtfed kernel: Swap cache stats: add 1236041, delete
1217294, find 52896/69574
Nov 28 06:55:01 virtfed kernel: Free swap  = 0kB
Nov 28 06:55:01 virtfed kernel: Total swap = 4194296kB
Nov 28 06:55:01 virtfed kernel: 3145727 pages RAM
Nov 28 06:55:01 virtfed kernel: 63020 pages reserved
Nov 28 06:55:01 virtfed kernel: 76044 pages shared
Nov 28 06:55:01 virtfed kernel: 3053641 pages non-shared
Nov 28 06:55:01 virtfed kernel: virbr0: port 3(vnet4) entering disabled state
Nov 28 06:55:01 virtfed kernel: device vnet4 left promiscuous mode
Nov 28 06:55:01 virtfed kernel: virbr0: port 3(vnet4) entering disabled state

About 1 hour and a half after, at 08:14 I get the same for the other 5.4 vm.
The same happens for other processes and the first 2 guests, taht are
configured to auto-start and they do so...

yesterday I restart the other 2 guests and this morning I have:
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2904 root      19  -1 10.6g 9.8g 2712 S  0.0 83.6  16:12.42 libvirtd

  [root virtfed ~]# ps -eo rss,pid,cmd|egrep libvirtd|egrep -v egrep
10309532 2904 libvirtd --daemon --listen

I have libvirtd as a service inside rhcs, so that every 30 seconds I have
rgmanager doing a check for its status:
Dec  1 10:56:39 virtfed bash[19916]: Executing /etc/init.d/libvirtd status

Don't know if this can influence the memory leak we are experiencing with
libvirtd....
Comment 32 Anders 2009-12-01 05:01:31 EST
Created attachment 375011 [details]
valgrind log from overnight run with libvirt.

I ran overnight
# valgrind --log-file=libvirtd.log --leak-check=full -v --time-stamp=yes /usr/sbin/libvirtd
and the logfile is attached. I'd had three VMs running and I had virt-manager started but done nothing with it. By about 10pm, everything was still okay, and valgrind/libvirtd took about 340MB. By 5am this morning, OOM killer started up and killed off various things (my Satellite VM having 3GB RAM allocated was one of the things gone).

If /var/log/messages from last night and onwards would help, I can provide that too. By the time I got to my system at 9am this morning, it took 30 minutes to get in to it so that I could finally kill valgrind/libvirtd, at which point its virt size was 8718MB and RES was 6.8GB according to top. (On a 8GB RAM, 2GB Swap system...)

Is there any other information, like /proc/.../maps that would help?

/Anders
Comment 33 Anders 2009-12-01 05:04:59 EST
Re comment#31, I ran a plain F11 system, then preupgraded to F12, so I've not gone via the "rawvirt on F11" step.
Comment 34 Daniel Berrange 2009-12-01 07:42:26 EST
After discussion on #virt IRC, it appears that the virt-manager monitoring may be a red-herring. There is a cluster-suite resource 'vm.sh' that is issuing 'virsh domstate $GUEST'  on a very frequent basis for all guests on a host. Since each invocation of virsh opens a libvirt connection, this will trigger the netcf memory leak bugs. With virsh being run so frequently, this will easily account for the huge memory usage of libvirtd please.

Please apply the following update ASAP, and restart the libvirtd daemon after installing it.

https://admin.fedoraproject.org/updates/netcf-0.1.5-1.fc12
Comment 35 Gianluca Cecchi 2009-12-01 09:39:59 EST
I don't think this is the (only at least) solution.
In fact on my system I have no netcf at all.
[root@virtfed ~]# rpm -q netcf
package netcf is not installed

My guests' logic is outside rhcs checks. Part of my cluster.conf is this:

                <service domain="DRBDNODE1" autostart="1" name="DRBDNODE1">
                <script ref="CLVMD">
                        <script ref="LIBVIRTD"/>
                </script>
                </service>

So in my case rgmanager is only doing "service libvirtd status" every 30 seconds.
I noticed this:
1) service libvirtd restart
2) Just after this I run this loop
[root@virtfed ~]# while true
> do
> ps -eo rss,pid,cmd|egrep libvirtd|egrep -v egrep
> sleep 30
> done

At the beginning I get
92948 14049 libvirtd --daemon --listen
97332 14049 libvirtd --daemon --listen
100408 14049 libvirtd --daemon --listen
97596 14049 libvirtd --daemon --listen
100148 14049 libvirtd --daemon --listen
100424 14049 libvirtd --daemon --listen

3) ssh my host with -X switch and run virt-manager and open 2 graphical consoles  
suddenly I get libvirtd memory grow about 3,5MB every 30 seconds
Now, after 2 hours and a half I have
1061572 14049 libvirtd --daemon --listen

4) I close virt-manager, but memory consumed by libvirtd still grows and grows every 30 seconds....

1077148 14049 libvirtd --daemon --listen
1079716 14049 libvirtd --daemon --listen
1082800 14049 libvirtd --daemon --listen
1085988 14049 libvirtd --daemon --listen
1089212 14049 libvirtd --daemon --listen
Comment 36 Daniel Berrange 2009-12-01 09:46:49 EST
You must have  netcf-libs  installed at the very least, since libvirtd links against it.
Comment 37 Anders 2009-12-01 09:53:43 EST
I've grabbed and installed the koji build of netcf-libs and I'm running with that at the moment. After an hour or so, libvirtd is still only taking up 20-30MB RES. Will push it a bit further and will start up virt-manager too, but I should know in about 5-6 hours if the main issue is gone.
Comment 38 Gianluca Cecchi 2009-12-01 10:01:36 EST
Ok, yes; I have
netcf-libs-0.1.4-1.fc12.x86_64
Updated to netcf-libs-0.1.5-1.fc12.x86_64.rpm
After restarting libvirtd and opening virt-manager and two consoles I get this
at the moment:
 7472 13736 libvirtd --daemon --listen
29652 13736 libvirtd --daemon --listen
43048 13736 libvirtd --daemon --listen
35856 13736 libvirtd --daemon --listen
24712 13736 libvirtd --daemon --listen
27468 13736 libvirtd --daemon --listen
29316 13736 libvirtd --daemon --listen
38876 13736 libvirtd --daemon --listen
40788 13736 libvirtd --daemon --listen
46424 13736 libvirtd --daemon --listen
46424 13736 libvirtd --daemon --listen
43564 13736 libvirtd --daemon --listen
43912 13736 libvirtd --daemon --listen

Let's see how and if it grows indefinitely...
Comment 39 Daniel Berrange 2009-12-01 10:08:55 EST
FYI, if anyone still sees a major, unbounded, memory leak after applying the netcf-libs-0.1.5  update, please collect some libvirtd logging

  service libvirtd stop
  edi /etc/libvirt/libvirtd.conf to set

    log_filters="1:libvirt"
    log_outputs="1:file:/var/log/libvirtd.log"

  service libvirtd start
 

then, do whatever you need todo to make the leak occur & let it run for a while, then attach the resulting /var/log/libvirtd.log to this bug


NB, it is normal to see libvirtd's memory usage flucuate +/-  20 MB  during use over a few minutes - the key is whether it grows without bound over a longer time
Comment 40 Anders 2009-12-02 03:22:35 EST
Have run overnight in the same circumstances that yesterday morning saw my system pretty much useless but this morning, libvirtd is sitting pretty at ~30MB RES memory still.

I will conclude that the problem I was seeing is alleviated with the netcf-libs-0.1.5 package.

Many thanks for the quick fix Daniel.

/Anders
Comment 41 Daniel Berrange 2009-12-02 05:33:22 EST
I'm making this bug depend on the 2 netcf fixes, and once those bugs close, I will close this one too.
Comment 42 Gianluca Cecchi 2009-12-04 11:19:08 EST
I confirm it is ok also in my environment with netcf-libs-0.1.5-1.fc12.x86_64.
Host has 12Gb of ram.
I've 5 guests running with mem configured respectively for 1024, 1024, 4096, 3512, 3512 Mb each.
virt-manager running and several vnc connections
rhcs running, so "service libvirtd status" running every 30 seconds
Now libvirtd up since more than 3 days and at about 30Mb too
ps -eo rss,pid,cmd|egrep libvirtd|egrep -v egrep
gives
30484 13736 libvirtd --daemon --listen

Also, ksm/ksmtuned are doing a great job, with now about 100k pages shared!

Thanks,
Gianluca
Comment 43 Cole Robinson 2010-05-10 16:16:23 EDT
Those netcf bugs are closed, so closing this as CURRENT_RELEASE.