Description of problem: kernel complaining about using smp_processor_id() in preemptible. [ 103.379280] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: dsm_sa_snmpd/5735 [ 103.379285] caller is native_sched_clock+0x49/0x90 [ 103.379287] CPU: 0 PID: 5735 Comm: dsm_sa_snmpd Not tainted 3.10.0-327.rt56.183.el6rt.x86_64.debug #1 [ 103.379288] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 2.2.10 11/09/2010 [ 103.379291] ffff8802f615bfd8 ffff8802f615b9e8 ffffffff81621b34 ffff8802f615ba08 [ 103.379292] ffffffff812f3394 000000b16dc6b696 ffff8805f6dfd200 ffff8802f615ba28 [ 103.379294] ffffffff8100cc59 0000000000000800 ffff88056f828000 ffff8802f615ba38 [ 103.379294] Call Trace: [ 103.379302] [<ffffffff81621b34>] dump_stack+0x19/0x1d [ 103.379306] [<ffffffff812f3394>] debug_smp_processor_id+0x124/0x150 [ 103.379308] [<ffffffff8100cc59>] native_sched_clock+0x49/0x90 [ 103.379311] [<ffffffff810a4a75>] local_clock+0x25/0x30 [ 103.379316] [<ffffffff815a9238>] tcp_transmit_skb+0x6d8/0x810 [ 103.379317] [<ffffffff815aae99>] tcp_write_xmit+0x209/0x7d0 [ 103.379320] [<ffffffff811845a0>] ? might_fault+0x40/0x90 [ 103.379322] [<ffffffff815aba52>] __tcp_push_pending_frames+0x32/0xd0 [ 103.379323] [<ffffffff811845a0>] ? might_fault+0x40/0x90 [ 103.379326] [<ffffffff8159880e>] tcp_push+0xee/0x120 [ 103.379327] [<ffffffff8159bfa5>] tcp_sendmsg+0x115/0xc90 [ 103.379330] [<ffffffff815ca303>] inet_sendmsg+0xc3/0x100 [ 103.379331] [<ffffffff815ca240>] ? inet_recvmsg+0x100/0x100 [ 103.379335] [<ffffffff81523f06>] sock_sendmsg+0xa6/0xd0 [ 103.379340] [<ffffffff811e3547>] ? fget_light+0xe7/0x150 [ 103.379342] [<ffffffff811e34a0>] ? fget_light+0x40/0x150 [ 103.379343] [<ffffffff81524067>] SYSC_sendto+0x137/0x190 [ 103.379347] [<ffffffff810cbfdd>] ? trace_hardirqs_on+0xd/0x10 [ 103.379350] [<ffffffff8115a993>] ? context_tracking_user_exit+0x13/0x20 [ 103.379353] [<ffffffff810115a9>] ? syscall_trace_enter+0x219/0x270 [ 103.379355] [<ffffffff815240ce>] SyS_sendto+0xe/0x10 [ 103.379358] [<ffffffff8162e7c1>] tracesys+0xdd/0xe2 Version-Release number of selected component (if applicable): 3.10.0-327.rt56.183.el6rt.x86_64.debug How reproducible: I asked the customer of the BZ1376184 to reproduce the problem using the debug kernel, and we saw these messages. It seems that it can be caused just by sending TCP packages. Actual results: kernel BUG messages. Expected results: No BUG messages. Additional info: vmcore at: http://file.rdu.redhat.com/~daolivei/SFDC01695456/vmcore
This issue can be fixed by this upstream commit: 5dd12c21 sched/clock, x86: Use mul_u64_u32_shr() for native_sched_clock()
The problem was observed just by booting *the debug kernel* on a customer's box. I think HTTP/SSH would be enough. Have you tried it? -- Daniel
(In reply to Daniel Bristot de Oliveira from comment #7) > The problem was observed just by booting *the debug kernel* on a customer's > box. I think HTTP/SSH would be enough. Have you tried it? > > -- Daniel regarding reproducer it is enough just boot box in lab. lab controller sends lot of packets to test machine: : beah-beaker-bac/11050 [ 68.579577] caller is native_sched_clock+0x49/0x90 [ 68.579578] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1 [ 68.579579] Hardware name: Hewlett-Packard /3396, BIOS K01 v02.51 06/15/2012 [ 68.579581] ffff880068b23fd8 ffff880068b23ca8 ffffffff81623353 ffff880068b23cc8 [ 68.579583] ffffffff812f4494 0000007d4e08197c ffff8800643b5800 ffff880068b23ce8 [ 68.579584] ffffffff8100cc89 ffff880068b23d57 ffff8800679e8000 ffff880068b23cf8 [ 68.579584] Call Trace: [ 68.579588] [<ffffffff81623353>] dump_stack+0x19/0x1e [ 68.579592] [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150 [ 68.579593] [<ffffffff8100cc89>] native_sched_clock+0x49/0x90 [ 68.579595] [<ffffffff810a61e5>] local_clock+0x25/0x30 [ 68.579598] [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810 [ 68.579600] [<ffffffff8152e258>] ? __alloc_skb+0xa8/0x1e0 [ 68.579601] [<ffffffff815ab7b1>] tcp_connect+0x271/0x2c0 [ 68.579603] [<ffffffff815b1056>] tcp_v4_connect+0x346/0x460 [ 68.579605] [<ffffffff815ca24d>] __inet_stream_connect+0x15d/0x1f0 [ 68.579606] [<ffffffff815ca317>] ? inet_stream_connect+0x37/0x70 [ 68.579607] [<ffffffff815ca328>] inet_stream_connect+0x48/0x70 [ 68.579610] [<ffffffff81524bea>] SYSC_connect+0xea/0x110 [ 68.579613] [<ffffffff811e4d70>] ? fget_light+0x40/0x150 [ 68.579615] [<ffffffff8162fe17>] ? sysret_check+0x1b/0x56 [ 68.579619] [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0 [ 68.579622] [<ffffffff812e914e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 68.579623] [<ffffffff81524c1e>] SyS_connect+0xe/0x10 [ 68.579624] [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b [ 68.580090] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050 [ 68.580093] caller is native_sched_clock+0x49/0x90 [ 68.580094] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1 [ 68.580094] Hardware name: Hewlett-Packard /3396, BIOS K01 v02.51 06/15/2012 [ 68.580096] ffff880068b23fd8 ffff880068b239e8 ffffffff81623353 ffff880068b23a08 [ 68.580097] ffffffff812f4494 0000007d4e22fb74 ffff8800643b4c00 ffff880068b23a28 [ 68.580099] ffffffff8100cc89 0000000000000800 ffff8800679e8000 ffff880068b23a38 [ 68.580099] Call Trace: [ 68.580101] [<ffffffff81623353>] dump_stack+0x19/0x1e [ 68.580103] [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150 [ 68.580104] [<ffffffff8100cc89>] native_sched_clock+0x49/0x90 [ 68.580105] [<ffffffff810a61e5>] local_clock+0x25/0x30 [ 68.580107] [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810 [ 68.580109] [<ffffffff8116809a>] ? __alloc_pages_nodemask+0x15a/0x330 [ 68.580111] [<ffffffff815ac689>] tcp_write_xmit+0x209/0x7d0 [ 68.580113] [<ffffffff81185f60>] ? might_fault+0x40/0x90 [ 68.580114] [<ffffffff815ad242>] __tcp_push_pending_frames+0x32/0xd0 [ 68.580115] [<ffffffff81185f60>] ? might_fault+0x40/0x90 [ 68.580116] [<ffffffff81599f8e>] tcp_push+0xee/0x120 [ 68.580118] [<ffffffff8159d725>] tcp_sendmsg+0x115/0xc90 [ 68.580119] [<ffffffff815cb9d3>] inet_sendmsg+0xc3/0x100 [ 68.580120] [<ffffffff815cb910>] ? inet_recvmsg+0x100/0x100 [ 68.580122] [<ffffffff81525166>] sock_sendmsg+0xa6/0xd0 [ 68.580124] [<ffffffff811e4e17>] ? fget_light+0xe7/0x150 [ 68.580126] [<ffffffff811e4d70>] ? fget_light+0x40/0x150 [ 68.580135] [<ffffffff81625a18>] ? rt_mutex_slowunlock+0x78/0x90 [ 68.580143] [<ffffffff815252c7>] SYSC_sendto+0x137/0x190 [ 68.580149] [<ffffffff81625a3e>] ? rt_mutex_unlock+0xe/0x10 [ 68.580152] [<ffffffff8162b8c6>] ? __do_page_fault+0x246/0x4d0 [ 68.580154] [<ffffffff8162818d>] ? retint_swapgs+0xe/0x13 [ 68.580156] [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0 [ 68.580158] [<ffffffff8152532e>] SyS_sendto+0xe/0x10 [ 68.580159] [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b [ 68.589646] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050 [ 68.589648] caller is native_sched_clock+0x49/0x90 [ 68.589649] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1 [ 68.589650] Hardware name: Hewlett-Packard /3396, BIOS K01 v02.51 06/15/2012 [ 68.589652] ffff880068b23fd8 ffff880068b23d28 ffffffff81623353 ffff880068b23d48 [ 68.589653] ffffffff812f4494 0000007d501168d8 ffff8800643b4e00 ffff880068b23d68 [ 68.589654] ffffffff8100cc89 ffff8800679e8000 ffff8800679e8000 ffff880068b23d78 [ 68.589654] Call Trace: [ 68.589656] [<ffffffff81623353>] dump_stack+0x19/0x1e [ 68.589658] [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150 [ 68.589660] [<ffffffff8100cc89>] native_sched_clock+0x49/0x90 [ 68.589661] [<ffffffff810a61e5>] local_clock+0x25/0x30 [ 68.589662] [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810 [ 68.589666] [<ffffffff811b1de2>] ? __kmalloc_node_track_caller+0x82/0x230 [ 68.589667] [<ffffffff815ac689>] tcp_write_xmit+0x209/0x7d0 [ 68.589669] [<ffffffff8152dccb>] ? __kmalloc_reserve+0x3b/0xa0 [ 68.589670] [<ffffffff815ad242>] __tcp_push_pending_frames+0x32/0xd0 [ 68.589672] [<ffffffff815ad354>] tcp_send_fin+0x74/0x1b0 [ 68.589673] [<ffffffff8159afc7>] tcp_shutdown+0x67/0x70 [ 68.589674] [<ffffffff815c9dd8>] inet_shutdown+0xb8/0x140 [ 68.589675] [<ffffffff815247fd>] SyS_shutdown+0x7d/0x90 [ 68.589677] [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b [ 68.594096] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050 [ 68.594099] caller is native_sched_clock+0x49/0x90 [ 68.594100] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1 [ 68.594100] Hardware name: Hewlett-Packard /3396, BIOS K01 v02.51 06/15/2012 [ 68.594102] ffff880068b23fd8 ffff880068b23ca8 ffffffff81623353 ffff880068b23cc8 [ 68.594103] ffffffff812f4494 0000007d50f7aa56 ffff8800643b4400 ffff880068b23ce8 [ 68.594105] ffffffff8100cc89 ffff880068b23d57 ffff8800679ec100 ffff880068b23cf8 [ 68.594105] Call Trace: [ 68.594107] [<ffffffff81623353>] dump_stack+0x19/0x1e [ 68.594109] [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150 [ 68.594110] [<ffffffff8100cc89>] native_sched_clock+0x49/0x90 [ 68.594111] [<ffffffff810a61e5>] local_clock+0x25/0x30 [ 68.594113] [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810 [ 68.594114] [<ffffffff8152e258>] ? __alloc_skb+0xa8/0x1e0 [ 68.594116] [<ffffffff815ab7b1>] tcp_connect+0x271/0x2c0 [ 68.594117] [<ffffffff815b1056>] tcp_v4_connect+0x346/0x460 [ 68.594119] [<ffffffff815ca24d>] __inet_stream_connect+0x15d/0x1f0 [ 68.594120] [<ffffffff815ca317>] ? inet_stream_connect+0x37/0x70 [ 68.594121] [<ffffffff815ca328>] inet_stream_connect+0x48/0x70 [ 68.594122] [<ffffffff81524bea>] SYSC_connect+0xea/0x110 [ 68.594124] [<ffffffff811e4d70>] ? fget_light+0x40/0x150 [ 68.594125] [<ffffffff8162fe17>] ? sysret_check+0x1b/0x56 [ 68.594127] [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0 [ 68.594129] [<ffffffff812e914e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 68.594130] [<ffffffff81524c1e>] SyS_connect+0xe/0x10 [ 68.594131] [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b [ 68.594523] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050 [ 68.594525] caller is native_sched_clock+0x49/0x90 [ 68.594526] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1 [ 68.594527] Hardware name: Hewlett-Packard /3396, BIOS K01 v02.51 06/15/2012 [ 68.594528] ffff880068b23fd8 ffff880068b239e8 ffffffff81623353 ffff880068b23a08 [ 68.594530] ffffffff812f4494 0000007d510dc470 ffff8800643b5400 ffff880068b23a28 [ 68.594531] ffffffff8100cc89 0000000000000800 ffff8800679ec100 ffff880068b23a38 [ 68.594531] Call Trace: [ 68.594533] [<ffffffff81623353>] dump_stack+0x19/0x1e [ 68.594534] [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150 [ 68.594536] [<ffffffff8100cc89>] native_sched_clock+0x49/0x90 [ 68.594537] [<ffffffff810a61e5>] local_clock+0x25/0x30 [ 68.594538] [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810 [ 68.594540] [<ffffffff815ac689>] tcp_write_xmit+0x209/0x7d0 [ 68.594541] [<ffffffff81185f60>] ? might_fault+0x40/0x90 [ 68.594543] [<ffffffff815ad242>] __tcp_push_pending_frames+0x32/0xd0 [ 68.594544] [<ffffffff81185f60>] ? might_fault+0x40/0x90 [ 68.594545] [<ffffffff81599f8e>] tcp_push+0xee/0x120 [ 68.594546] [<ffffffff8159d725>] tcp_sendmsg+0x115/0xc90 [ 68.594547] [<ffffffff815cb9d3>] inet_sendmsg+0xc3/0x100 [ 68.594548] [<ffffffff815cb910>] ? inet_recvmsg+0x100/0x100 [ 68.594550] [<ffffffff81525166>] sock_sendmsg+0xa6/0xd0 [ 68.594552] [<ffffffff811e4e17>] ? fget_light+0xe7/0x150 [ 68.594553] [<ffffffff811e4d70>] ? fget_light+0x40/0x150 [ 68.594555] [<ffffffff815252c7>] SYSC_sendto+0x137/0x190 [ 68.594556] [<ffffffff8162fe17>] ? sysret_check+0x1b/0x56 [ 68.594558] [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0 [ 68.594559] [<ffffffff8152532e>] SyS_sendto+0xe/0x10 [ 68.594561] [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b [ 68.680495] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050 [ 68.680497] caller is native_sched_clock+0x49/0x90 [ 68.680499] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1 [ 68.680499] Hardware name: Hewlett-Packard /3396, BIOS K01 v02.51 06/15/2012 [ 68.680501] ffff880068b23fd8 ffff880068b23d28 ffffffff81623353 ffff880068b23d48 [ 68.680502] ffffffff812f4494 0000007d626dc9f0 ffff8800643b5e00 ffff880068b23d68
[root@hp-8300e-01 ~]# head -n 50 dmesg.diff --- dmesg-3.10.0-327.rt56.199.el6rt.x86_64.debug.notime 2017-01-06 07:29:50.803706519 -0500 +++ dmesg-3.10.0-514.rt56.210.el6rt.x86_64.debug.notime 2017-01-06 07:33:57.234605156 -0500 @@ -1,1857 +1,31 @@ - input: HDA Intel PCH Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input4 - input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input5 - input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6 - input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7 - input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8 + input: HDA Intel PCH Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input3 + input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input4 + input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input5 + input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6 + input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7 + input: PC Speaker as /devices/platform/pcspkr/input/input8 + sd 0:0:0:0: Attached scsi generic sg0 type 0 + sr 2:0:0:0: Attached scsi generic sg1 type 5 + ACPI Warning: SystemIO range 0x000000000000f040-0x000000000000f05f conflicts with OpRegion 0x000000000000f040-0x000000000000f04f (\_SB_.PCI0.SBUS.SMBI) (20130517/utaddress-254) + ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver input: HP WMI hotkeys as /devices/virtual/input/input9 iTCO_vendor_support: vendor-support=0 iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 iTCO_wdt: Found a Panther Point TCO device (Version=2, TCOBASE=0x0460) iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) ppdev: user-space parallel port driver + Loaded the dummy scsi_wait_scan module + unloading the dummy scsi_wait_scan module + Loaded the dummy scsi_wait_scan module + unloading the dummy scsi_wait_scan module EXT4-fs (dm-0): re-mounted. Opts: (null) EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) - SELinux: initialized (dev sda1, type ext4), uses xattr EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) - SELinux: initialized (dev dm-2, type ext4), uses xattr Adding 3735548k swap on /dev/mapper/vg_hp8300e01-lv_swap. Priority:-1 extents:1 across:3735548k FS - SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts NET: Registered protocol family 10 e1000e 0000:00:19.0: irq 29 for MSI/MSI-X e1000e 0000:00:19.0: irq 29 for MSI/MSI-X IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready - type=1305 audit(1483703888.868:4): audit_pid=3345 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1 - BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/3868 - caller is native_sched_clock+0x49/0x90 - CPU: 0 PID: 3868 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1 - Hardware name: Hewlett-Packard /3396, BIOS K01 v02.51 06/15/2012 - ffff880067a57fd8 ffff880067a57ca8 ffffffff81623353 ffff880067a57cc8 - ffffffff812f4494 00000065af7f4b5e ffff88006455de00 ffff880067a57ce8 - ffffffff8100cc89 ffff880067a57d57 ffff880068ce0000 ffff880067a57cf8 - Call Trace:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0113.html