Bug 1453145 - Brick Multiplexing:dmesg shows request_sock_TCP: Possible SYN flooding on port 49152
Summary: Brick Multiplexing:dmesg shows request_sock_TCP: Possible SYN flooding on por...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core
Version: rhgs-3.3
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: RHGS 3.3.0
Assignee: Mohammed Rafi KC
QA Contact: nchilaka
URL:
Whiteboard: brick-multiplexing
Depends On:
Blocks: 1417151 1456405 1463517
TreeView+ depends on / blocked
 
Reported: 2017-05-22 09:21 UTC by nchilaka
Modified: 2018-01-20 09:00 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.8.4-29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1456405 (view as bug list)
Environment:
Last Closed: 2017-09-21 04:43:23 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2774 normal SHIPPED_LIVE glusterfs bug fix and enhancement update 2017-09-21 08:16:29 UTC

Description nchilaka 2017-05-22 09:21:41 UTC
Description of problem:
========================
In my brick mux setup, which hosts about 40+ 1x3 volumes oversame brick pids,
I ran IOs over a day for about 10 volumes from 10 dedicated clients.

However I noticed one of the server dmesg which had below backtrace

[Mon May 22 10:32:47 2017] Kprobes globally unoptimized
[Mon May 22 10:32:47 2017] stap_f2381505060ae6310863f02aed9a39ef_22947: loading out-of-tree module taints kernel.
[Mon May 22 10:32:47 2017] stap_f2381505060ae6310863f02aed9a39ef_22947: module verification failed: signature and/or required key missing - tainting kernel
[Mon May 22 10:32:48 2017] stap_f2381505060ae6310863f02aed9a39ef_22947: systemtap: 3.0/0.166, base: ffffffffa0608000, memory: 144data/64text/83ctx/2058net/50alloc kb, probes: 1
[Mon May 22 10:34:15 2017] stap_daf3ce6b146bb54d562d096209111fa0_23295: systemtap: 3.0/0.166, base: ffffffffa063e000, memory: 145data/68text/83ctx/2058net/50alloc kb, probes: 1
[Mon May 22 10:35:06 2017] stap_283ca8e5669d13a9ef14e3bb2b601f89_23574: systemtap: 3.0/0.166, base: ffffffffa0675000, memory: 145data/68text/86ctx/2058net/50alloc kb, probes: 1
[Mon May 22 10:40:56 2017] staprun: page allocation failure: order:4, mode:0x10c0d0
[Mon May 22 10:40:56 2017] CPU: 5 PID: 24031 Comm: staprun Tainted: G           OE  ------------   3.10.0-514.10.2.el7.x86_64 #1
[Mon May 22 10:40:56 2017] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[Wed May 17 18:39:41 2017] intel_powerclamp: No package C-state available
[Mon May 22 10:40:56 2017]  000000000010c0d0
[Mon May 22 10:40:56 2017]  00000000eb8721f0 ffff88012cc37988 ffffffff816863ef
[Mon May 22 10:40:56 2017]  ffff88012cc37a18 ffffffff81186ba0 0000000000000000 00000000ffffffff
[Mon May 22 10:40:56 2017]  fffffffffffffff0 0010c0d000000000 ffff88012cc379e8 00000000eb8721f0
[Mon May 22 10:40:56 2017] Call Trace:
[Mon May 22 10:40:56 2017]  [<ffffffff816863ef>] dump_stack+0x19/0x1b
[Mon May 22 10:40:56 2017]  [<ffffffff81186ba0>] warn_alloc_failed+0x110/0x180
[Mon May 22 10:40:56 2017]  [<ffffffff81681f83>] __alloc_pages_slowpath+0x6b7/0x725
[Mon May 22 10:40:56 2017]  [<ffffffff8118b155>] __alloc_pages_nodemask+0x405/0x420
[Mon May 22 10:40:56 2017]  [<ffffffff811cf30a>] alloc_pages_current+0xaa/0x170
[Mon May 22 10:40:56 2017]  [<ffffffff81185a7e>] __get_free_pages+0xe/0x50
[Mon May 22 10:40:56 2017]  [<ffffffff811dabae>] kmalloc_order_trace+0x2e/0xa0
[Mon May 22 10:40:56 2017]  [<ffffffff8113c723>] relay_open+0x63/0x2c0
[Mon May 22 10:40:56 2017]  [<ffffffffa0612840>] ? _stp_linenumber_lookup+0x20/0x20 [stap_e6774b0dd49a63e06112bfeaa9167888_24031]
[Mon May 22 10:40:56 2017]  [<ffffffffa0609b57>] _stp_transport_data_fs_init+0xc7/0x180 [stap_e6774b0dd49a63e06112bfeaa9167888_24031]
[Mon May 22 10:40:56 2017]  [<ffffffff812156d0>] ? dput+0x20/0xd0
[Mon May 22 10:40:56 2017]  [<ffffffff8121d48a>] ? __get_fs_type+0x3a/0x60
[Mon May 22 10:40:56 2017]  [<ffffffffa0612969>] init_module+0x129/0x950 [stap_e6774b0dd49a63e06112bfeaa9167888_24031]
[Mon May 22 10:40:56 2017]  [<ffffffffa0612840>] ? _stp_linenumber_lookup+0x20/0x20 [stap_e6774b0dd49a63e06112bfeaa9167888_24031]
[Mon May 22 10:40:56 2017]  [<ffffffff810020e8>] do_one_initcall+0xb8/0x230
[Mon May 22 10:40:56 2017]  [<ffffffff81100488>] load_module+0x22c8/0x2930
[Mon May 22 10:40:56 2017]  [<ffffffff8133d6e0>] ? ddebug_proc_write+0xf0/0xf0
[Mon May 22 10:40:56 2017]  [<ffffffff81100bb5>] SyS_init_module+0xc5/0x110
[Mon May 22 10:40:56 2017]  [<ffffffff81696a09>] system_call_fastpath+0x16/0x1b
[Mon May 22 10:40:56 2017] Mem-Info:
[Mon May 22 10:40:56 2017] active_anon:537547 inactive_anon:6033 isolated_anon:0
 active_file:329546 inactive_file:332986 isolated_file:0
 unevictable:6524 dirty:355 writeback:0 unstable:0
 slab_reclaimable:333111 slab_unreclaimable:119891
 mapped:9475 shmem:11538 pagetables:4152 bounce:0
 free:67311 free_pcp:0 free_cma:0
[Mon May 22 10:40:56 2017] Node 0 DMA free:15908kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[Mon May 22 10:40:56 2017] lowmem_reserve[]: 0 2816 7813 7813
[Mon May 22 10:40:56 2017] Node 0 DMA32 free:89160kB min:24308kB low:30384kB high:36460kB active_anon:1059552kB inactive_anon:3796kB active_file:413284kB inactive_file:409652kB unevictable:8908kB isolated(anon):0kB isolated(file):0kB present:3129336kB managed:2884596kB mlocked:8908kB dirty:312kB writeback:0kB mapped:9644kB shmem:7988kB slab_reclaimable:639436kB slab_unreclaimable:103088kB kernel_stack:10096kB pagetables:4284kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[Mon May 22 10:40:56 2017] lowmem_reserve[]: 0 0 4996 4996
[Mon May 22 10:40:56 2017] Node 0 Normal free:164176kB min:43136kB low:53920kB high:64704kB active_anon:1090636kB inactive_anon:20336kB active_file:904900kB inactive_file:922292kB unevictable:17188kB isolated(anon):0kB isolated(file):0kB present:5251072kB managed:5115920kB mlocked:17188kB dirty:1108kB writeback:0kB mapped:28256kB shmem:38164kB slab_reclaimable:693008kB slab_unreclaimable:376476kB kernel_stack:19152kB pagetables:12324kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[Mon May 22 10:40:56 2017] lowmem_reserve[]: 0 0 0 0
[Mon May 22 10:40:56 2017] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[Mon May 22 10:40:56 2017] Node 0 DMA32: 9232*4kB (UEM) 3811*8kB (UEM) 1341*16kB (UEM) 20*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 89512kB
[Mon May 22 10:40:56 2017] Node 0 Normal: 24334*4kB (UEM) 3413*8kB (UEM) 1857*16kB (UEM) 292*32kB (UEM) 16*64kB (EM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 164720kB
[Mon May 22 10:40:56 2017] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Mon May 22 10:40:56 2017] 676567 total pagecache pages
[Mon May 22 10:40:56 2017] 0 pages in swap cache
[Mon May 22 10:40:56 2017] Swap cache stats: add 0, delete 0, find 0/0
[Mon May 22 10:40:56 2017] Free swap  = 0kB
[Mon May 22 10:40:56 2017] Total swap = 0kB
[Mon May 22 10:40:56 2017] 2099100 pages RAM
[Mon May 22 10:40:56 2017] 0 pages HighMem/MovableOnly
[Mon May 22 10:40:56 2017] 94994 pages reserved
[Mon May 22 10:41:18 2017] stap_ca16476c9707735de6b0448a9e43e59_24277: systemtap: 3.0/0.166, base: ffffffffa0641000, memory: 139data/48text/37ctx/2058net/50alloc kb, probes: 1
[Mon May 22 10:44:43 2017] stap_283ca8e5669d13a9ef14e3bb2b601f89_24407: systemtap: 3.0/0.166, base: ffffffffa0608000, memory: 145data/68text/86ctx/2058net/50alloc kb, probes: 1
[Mon May 22 10:45:28 2017] stap_ca16476c9707735de6b0448a9e43e59_24448: systemtap: 3.0/0.166, base: ffffffffa063f000, memory: 139data/48text/37ctx/2058net/50alloc kb, probes: 1
[Mon May 22 11:02:00 2017] TCP: request_sock_TCP: Possible SYN flooding on port 49152. Sending cookies.  Check SNMP counters.





Version-Release number of selected component (if applicable):
===========
3.8.4-25

Steps to Reproduce:
1.40+ volumes on brick mux setup all 1x3 vols
2.mounted 10 volumes on 10 different clients
3.performed IO over a day

Comment 8 Mohammed Rafi KC 2017-05-24 07:27:41 UTC
We can have a volume-set command to specify the backlog limits, but remember we need to configure kernal parameters like net.core.somaxconn, net.ipv4.tcp_max_syn_backlog .

Comment 11 Mohammed Rafi KC 2017-05-29 12:41:14 UTC
upstream patch : https://review.gluster.org/#/c/17411
                 https://review.gluster.org/#/c/17412

Comment 17 Atin Mukherjee 2017-06-15 05:21:51 UTC
one more downstream patch https://code.engineering.redhat.com/gerrit/#/c/109134/ to fix the max op-version is needed which is merged now. Moving this bug status back to MODIFIED as the fix will work from next build only.

Comment 19 Mohammed Rafi KC 2017-07-14 09:02:48 UTC
We have addressed the doc updates using a separate bug 1461430

Comment 21 Mohammed Rafi KC 2017-07-28 13:35:16 UTC
steps to test this bug :
1)Before creating the volumes and after setting multiplex on , set the nvolume option transport.listen-backlog 128 

There are two system level options that need to set (# sysctl -p net.core.somaxconn=$value , # sysctl -p net.ipv4.tcp_max_syn_backlog=$value)

I cannot recommend a value for this , it is based on system configuration and the application.

For now you can go with the default values

Comment 22 nchilaka 2017-08-03 09:31:56 UTC
on_qa validation:
tested with Same kind of workload(or even more) for almost 2 days 
I set the value of transport.listen-backlog to 128 as recommended by Rafi in Comment#21
left the system ctl values to default which were as below
net.core.somaxconn = 128
net.ipv4.tcp_max_syn_backlog = 256

have 40 1x3 vols with brick mux enabled and all bricks running on one fsd proc

had 10 volumes mounted on different clients each, and did linux untar for 50 times in each volume, for more than 1.5 days.

Didn't hit the syn flooding problem.

hence moving to verified


testverion:3.8.4-36 on rhel7.4

Comment 23 nchilaka 2017-08-03 12:03:32 UTC
command for sysctl option is sysctl net.ipv4.tcp_max_syn_backlog=$value (no -p)

Comment 25 errata-xmlrpc 2017-09-21 04:43:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774


Note You need to log in before you can comment on or make changes to this bug.