Bug 1378982

Summary:	BUG: using smp_processor_id() in preemptible
Product:	Red Hat Enterprise MRG	Reporter:	Daniel Bristot de Oliveira <daolivei>
Component:	realtime-kernel	Assignee:	Luis Claudio R. Goncalves <lgoncalv>
Status:	CLOSED ERRATA	QA Contact:	MRG Quality Engineering <mrgqe-bugs>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	2.5	CC:	bhu, daolivei, jkastner, lgoncalv, saime, williams
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: The RT patchset causes the kernel function smp_processor_id() to be called in atomic context from the function __cycles_2_ns(). Consequence: Console warnings that a sleeping lock is being acquired in atomic context. Fix: Apply upstream commit 5dd12c21 which changes the way the __cyles_2_ns() does a 64-bit multiplication, thus removing the need for a call to smp_processor_id(). Result: Operations with no console warnings due to calls to __cycles_2_ns() inside atomic context.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-01-17 18:01:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Daniel Bristot de Oliveira 2016-09-23 19:19:41 UTC

Description of problem:

kernel complaining about using smp_processor_id() in preemptible.

[  103.379280] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: dsm_sa_snmpd/5735
[  103.379285] caller is native_sched_clock+0x49/0x90
[  103.379287] CPU: 0 PID: 5735 Comm: dsm_sa_snmpd Not tainted 3.10.0-327.rt56.183.el6rt.x86_64.debug #1
[  103.379288] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 2.2.10 11/09/2010
[  103.379291]  ffff8802f615bfd8 ffff8802f615b9e8 ffffffff81621b34 ffff8802f615ba08
[  103.379292]  ffffffff812f3394 000000b16dc6b696 ffff8805f6dfd200 ffff8802f615ba28
[  103.379294]  ffffffff8100cc59 0000000000000800 ffff88056f828000 ffff8802f615ba38
[  103.379294] Call Trace:
[  103.379302]  [<ffffffff81621b34>] dump_stack+0x19/0x1d
[  103.379306]  [<ffffffff812f3394>] debug_smp_processor_id+0x124/0x150
[  103.379308]  [<ffffffff8100cc59>] native_sched_clock+0x49/0x90
[  103.379311]  [<ffffffff810a4a75>] local_clock+0x25/0x30
[  103.379316]  [<ffffffff815a9238>] tcp_transmit_skb+0x6d8/0x810
[  103.379317]  [<ffffffff815aae99>] tcp_write_xmit+0x209/0x7d0
[  103.379320]  [<ffffffff811845a0>] ? might_fault+0x40/0x90
[  103.379322]  [<ffffffff815aba52>] __tcp_push_pending_frames+0x32/0xd0
[  103.379323]  [<ffffffff811845a0>] ? might_fault+0x40/0x90
[  103.379326]  [<ffffffff8159880e>] tcp_push+0xee/0x120
[  103.379327]  [<ffffffff8159bfa5>] tcp_sendmsg+0x115/0xc90
[  103.379330]  [<ffffffff815ca303>] inet_sendmsg+0xc3/0x100
[  103.379331]  [<ffffffff815ca240>] ? inet_recvmsg+0x100/0x100
[  103.379335]  [<ffffffff81523f06>] sock_sendmsg+0xa6/0xd0
[  103.379340]  [<ffffffff811e3547>] ? fget_light+0xe7/0x150
[  103.379342]  [<ffffffff811e34a0>] ? fget_light+0x40/0x150
[  103.379343]  [<ffffffff81524067>] SYSC_sendto+0x137/0x190
[  103.379347]  [<ffffffff810cbfdd>] ? trace_hardirqs_on+0xd/0x10
[  103.379350]  [<ffffffff8115a993>] ? context_tracking_user_exit+0x13/0x20
[  103.379353]  [<ffffffff810115a9>] ? syscall_trace_enter+0x219/0x270
[  103.379355]  [<ffffffff815240ce>] SyS_sendto+0xe/0x10
[  103.379358]  [<ffffffff8162e7c1>] tracesys+0xdd/0xe2

Version-Release number of selected component (if applicable):
3.10.0-327.rt56.183.el6rt.x86_64.debug

How reproducible:

I asked the customer of the BZ1376184 to reproduce the problem using the debug kernel, and we saw these messages.
It seems that it can be caused just by sending TCP packages.

Actual results:
kernel BUG messages.

Expected results:
No BUG messages.

Additional info:
vmcore at: http://file.rdu.redhat.com/~daolivei/SFDC01695456/vmcore

Comment 1 Luis Claudio R. Goncalves 2016-10-04 12:05:05 UTC

This issue can be fixed by this upstream commit: 

   5dd12c21 sched/clock, x86: Use mul_u64_u32_shr() for native_sched_clock()

Comment 7 Daniel Bristot de Oliveira 2016-11-09 16:15:13 UTC

The problem was observed just by booting *the debug kernel* on a customer's box. I think HTTP/SSH would be enough. Have you tried it?

-- Daniel

Comment 10 Jiri Kastner 2017-01-06 10:46:41 UTC

(In reply to Daniel Bristot de Oliveira from comment #7)
> The problem was observed just by booting *the debug kernel* on a customer's
> box. I think HTTP/SSH would be enough. Have you tried it?
> 
> -- Daniel

regarding reproducer it is enough just boot box in lab. lab controller sends lot of packets to test machine:

: beah-beaker-bac/11050
[   68.579577] caller is native_sched_clock+0x49/0x90
[   68.579578] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1
[   68.579579] Hardware name: Hewlett-Packard  /3396, BIOS K01 v02.51 06/15/2012
[   68.579581]  ffff880068b23fd8 ffff880068b23ca8 ffffffff81623353 ffff880068b23cc8
[   68.579583]  ffffffff812f4494 0000007d4e08197c ffff8800643b5800 ffff880068b23ce8
[   68.579584]  ffffffff8100cc89 ffff880068b23d57 ffff8800679e8000 ffff880068b23cf8
[   68.579584] Call Trace:
[   68.579588]  [<ffffffff81623353>] dump_stack+0x19/0x1e
[   68.579592]  [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150
[   68.579593]  [<ffffffff8100cc89>] native_sched_clock+0x49/0x90
[   68.579595]  [<ffffffff810a61e5>] local_clock+0x25/0x30
[   68.579598]  [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810
[   68.579600]  [<ffffffff8152e258>] ? __alloc_skb+0xa8/0x1e0
[   68.579601]  [<ffffffff815ab7b1>] tcp_connect+0x271/0x2c0
[   68.579603]  [<ffffffff815b1056>] tcp_v4_connect+0x346/0x460
[   68.579605]  [<ffffffff815ca24d>] __inet_stream_connect+0x15d/0x1f0
[   68.579606]  [<ffffffff815ca317>] ? inet_stream_connect+0x37/0x70
[   68.579607]  [<ffffffff815ca328>] inet_stream_connect+0x48/0x70
[   68.579610]  [<ffffffff81524bea>] SYSC_connect+0xea/0x110
[   68.579613]  [<ffffffff811e4d70>] ? fget_light+0x40/0x150
[   68.579615]  [<ffffffff8162fe17>] ? sysret_check+0x1b/0x56
[   68.579619]  [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0
[   68.579622]  [<ffffffff812e914e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   68.579623]  [<ffffffff81524c1e>] SyS_connect+0xe/0x10
[   68.579624]  [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b
[   68.580090] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050
[   68.580093] caller is native_sched_clock+0x49/0x90
[   68.580094] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1
[   68.580094] Hardware name: Hewlett-Packard  /3396, BIOS K01 v02.51 06/15/2012
[   68.580096]  ffff880068b23fd8 ffff880068b239e8 ffffffff81623353 ffff880068b23a08
[   68.580097]  ffffffff812f4494 0000007d4e22fb74 ffff8800643b4c00 ffff880068b23a28
[   68.580099]  ffffffff8100cc89 0000000000000800 ffff8800679e8000 ffff880068b23a38
[   68.580099] Call Trace:
[   68.580101]  [<ffffffff81623353>] dump_stack+0x19/0x1e
[   68.580103]  [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150
[   68.580104]  [<ffffffff8100cc89>] native_sched_clock+0x49/0x90
[   68.580105]  [<ffffffff810a61e5>] local_clock+0x25/0x30
[   68.580107]  [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810
[   68.580109]  [<ffffffff8116809a>] ? __alloc_pages_nodemask+0x15a/0x330
[   68.580111]  [<ffffffff815ac689>] tcp_write_xmit+0x209/0x7d0
[   68.580113]  [<ffffffff81185f60>] ? might_fault+0x40/0x90
[   68.580114]  [<ffffffff815ad242>] __tcp_push_pending_frames+0x32/0xd0
[   68.580115]  [<ffffffff81185f60>] ? might_fault+0x40/0x90
[   68.580116]  [<ffffffff81599f8e>] tcp_push+0xee/0x120
[   68.580118]  [<ffffffff8159d725>] tcp_sendmsg+0x115/0xc90
[   68.580119]  [<ffffffff815cb9d3>] inet_sendmsg+0xc3/0x100
[   68.580120]  [<ffffffff815cb910>] ? inet_recvmsg+0x100/0x100
[   68.580122]  [<ffffffff81525166>] sock_sendmsg+0xa6/0xd0
[   68.580124]  [<ffffffff811e4e17>] ? fget_light+0xe7/0x150
[   68.580126]  [<ffffffff811e4d70>] ? fget_light+0x40/0x150
[   68.580135]  [<ffffffff81625a18>] ? rt_mutex_slowunlock+0x78/0x90
[   68.580143]  [<ffffffff815252c7>] SYSC_sendto+0x137/0x190
[   68.580149]  [<ffffffff81625a3e>] ? rt_mutex_unlock+0xe/0x10
[   68.580152]  [<ffffffff8162b8c6>] ? __do_page_fault+0x246/0x4d0
[   68.580154]  [<ffffffff8162818d>] ? retint_swapgs+0xe/0x13
[   68.580156]  [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0
[   68.580158]  [<ffffffff8152532e>] SyS_sendto+0xe/0x10
[   68.580159]  [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b
[   68.589646] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050
[   68.589648] caller is native_sched_clock+0x49/0x90
[   68.589649] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1
[   68.589650] Hardware name: Hewlett-Packard  /3396, BIOS K01 v02.51 06/15/2012
[   68.589652]  ffff880068b23fd8 ffff880068b23d28 ffffffff81623353 ffff880068b23d48
[   68.589653]  ffffffff812f4494 0000007d501168d8 ffff8800643b4e00 ffff880068b23d68
[   68.589654]  ffffffff8100cc89 ffff8800679e8000 ffff8800679e8000 ffff880068b23d78
[   68.589654] Call Trace:
[   68.589656]  [<ffffffff81623353>] dump_stack+0x19/0x1e
[   68.589658]  [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150
[   68.589660]  [<ffffffff8100cc89>] native_sched_clock+0x49/0x90
[   68.589661]  [<ffffffff810a61e5>] local_clock+0x25/0x30
[   68.589662]  [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810
[   68.589666]  [<ffffffff811b1de2>] ? __kmalloc_node_track_caller+0x82/0x230
[   68.589667]  [<ffffffff815ac689>] tcp_write_xmit+0x209/0x7d0
[   68.589669]  [<ffffffff8152dccb>] ? __kmalloc_reserve+0x3b/0xa0
[   68.589670]  [<ffffffff815ad242>] __tcp_push_pending_frames+0x32/0xd0
[   68.589672]  [<ffffffff815ad354>] tcp_send_fin+0x74/0x1b0
[   68.589673]  [<ffffffff8159afc7>] tcp_shutdown+0x67/0x70
[   68.589674]  [<ffffffff815c9dd8>] inet_shutdown+0xb8/0x140
[   68.589675]  [<ffffffff815247fd>] SyS_shutdown+0x7d/0x90
[   68.589677]  [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b
[   68.594096] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050
[   68.594099] caller is native_sched_clock+0x49/0x90
[   68.594100] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1
[   68.594100] Hardware name: Hewlett-Packard  /3396, BIOS K01 v02.51 06/15/2012
[   68.594102]  ffff880068b23fd8 ffff880068b23ca8 ffffffff81623353 ffff880068b23cc8
[   68.594103]  ffffffff812f4494 0000007d50f7aa56 ffff8800643b4400 ffff880068b23ce8
[   68.594105]  ffffffff8100cc89 ffff880068b23d57 ffff8800679ec100 ffff880068b23cf8
[   68.594105] Call Trace:
[   68.594107]  [<ffffffff81623353>] dump_stack+0x19/0x1e
[   68.594109]  [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150
[   68.594110]  [<ffffffff8100cc89>] native_sched_clock+0x49/0x90
[   68.594111]  [<ffffffff810a61e5>] local_clock+0x25/0x30
[   68.594113]  [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810
[   68.594114]  [<ffffffff8152e258>] ? __alloc_skb+0xa8/0x1e0
[   68.594116]  [<ffffffff815ab7b1>] tcp_connect+0x271/0x2c0
[   68.594117]  [<ffffffff815b1056>] tcp_v4_connect+0x346/0x460
[   68.594119]  [<ffffffff815ca24d>] __inet_stream_connect+0x15d/0x1f0
[   68.594120]  [<ffffffff815ca317>] ? inet_stream_connect+0x37/0x70
[   68.594121]  [<ffffffff815ca328>] inet_stream_connect+0x48/0x70
[   68.594122]  [<ffffffff81524bea>] SYSC_connect+0xea/0x110
[   68.594124]  [<ffffffff811e4d70>] ? fget_light+0x40/0x150
[   68.594125]  [<ffffffff8162fe17>] ? sysret_check+0x1b/0x56
[   68.594127]  [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0
[   68.594129]  [<ffffffff812e914e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[   68.594130]  [<ffffffff81524c1e>] SyS_connect+0xe/0x10
[   68.594131]  [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b
[   68.594523] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050
[   68.594525] caller is native_sched_clock+0x49/0x90
[   68.594526] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1
[   68.594527] Hardware name: Hewlett-Packard  /3396, BIOS K01 v02.51 06/15/2012
[   68.594528]  ffff880068b23fd8 ffff880068b239e8 ffffffff81623353 ffff880068b23a08
[   68.594530]  ffffffff812f4494 0000007d510dc470 ffff8800643b5400 ffff880068b23a28
[   68.594531]  ffffffff8100cc89 0000000000000800 ffff8800679ec100 ffff880068b23a38
[   68.594531] Call Trace:
[   68.594533]  [<ffffffff81623353>] dump_stack+0x19/0x1e
[   68.594534]  [<ffffffff812f4494>] debug_smp_processor_id+0x124/0x150
[   68.594536]  [<ffffffff8100cc89>] native_sched_clock+0x49/0x90
[   68.594537]  [<ffffffff810a61e5>] local_clock+0x25/0x30
[   68.594538]  [<ffffffff815aaa28>] tcp_transmit_skb+0x6d8/0x810
[   68.594540]  [<ffffffff815ac689>] tcp_write_xmit+0x209/0x7d0
[   68.594541]  [<ffffffff81185f60>] ? might_fault+0x40/0x90
[   68.594543]  [<ffffffff815ad242>] __tcp_push_pending_frames+0x32/0xd0
[   68.594544]  [<ffffffff81185f60>] ? might_fault+0x40/0x90
[   68.594545]  [<ffffffff81599f8e>] tcp_push+0xee/0x120
[   68.594546]  [<ffffffff8159d725>] tcp_sendmsg+0x115/0xc90
[   68.594547]  [<ffffffff815cb9d3>] inet_sendmsg+0xc3/0x100
[   68.594548]  [<ffffffff815cb910>] ? inet_recvmsg+0x100/0x100
[   68.594550]  [<ffffffff81525166>] sock_sendmsg+0xa6/0xd0
[   68.594552]  [<ffffffff811e4e17>] ? fget_light+0xe7/0x150
[   68.594553]  [<ffffffff811e4d70>] ? fget_light+0x40/0x150
[   68.594555]  [<ffffffff815252c7>] SYSC_sendto+0x137/0x190
[   68.594556]  [<ffffffff8162fe17>] ? sysret_check+0x1b/0x56
[   68.594558]  [<ffffffff810cd660>] ? trace_hardirqs_on_caller+0x20/0x1e0
[   68.594559]  [<ffffffff8152532e>] SyS_sendto+0xe/0x10
[   68.594561]  [<ffffffff8162fdf2>] system_call_fastpath+0x16/0x1b
[   68.680495] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/11050
[   68.680497] caller is native_sched_clock+0x49/0x90
[   68.680499] CPU: 3 PID: 11050 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1
[   68.680499] Hardware name: Hewlett-Packard  /3396, BIOS K01 v02.51 06/15/2012
[   68.680501]  ffff880068b23fd8 ffff880068b23d28 ffffffff81623353 ffff880068b23d48
[   68.680502]  ffffffff812f4494 0000007d626dc9f0 ffff8800643b5e00 ffff880068b23d68

Comment 11 Jiri Kastner 2017-01-06 12:36:45 UTC

[root@hp-8300e-01 ~]# head -n 50 dmesg.diff
--- dmesg-3.10.0-327.rt56.199.el6rt.x86_64.debug.notime 2017-01-06 07:29:50.803706519 -0500
+++ dmesg-3.10.0-514.rt56.210.el6rt.x86_64.debug.notime 2017-01-06 07:33:57.234605156 -0500
@@ -1,1857 +1,31 @@
- input: HDA Intel PCH Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input4
- input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input5
- input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6
- input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7
- input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8
+ input: HDA Intel PCH Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input3
+ input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input4
+ input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input5
+ input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6
+ input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7
+ input: PC Speaker as /devices/platform/pcspkr/input/input8
+ sd 0:0:0:0: Attached scsi generic sg0 type 0
+ sr 2:0:0:0: Attached scsi generic sg1 type 5
+ ACPI Warning: SystemIO range 0x000000000000f040-0x000000000000f05f conflicts with OpRegion 0x000000000000f040-0x000000000000f04f (\_SB_.PCI0.SBUS.SMBI) (20130517/utaddress-254)
+ ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
  input: HP WMI hotkeys as /devices/virtual/input/input9
  iTCO_vendor_support: vendor-support=0
  iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
  iTCO_wdt: Found a Panther Point TCO device (Version=2, TCOBASE=0x0460)
  iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
  ppdev: user-space parallel port driver
+ Loaded the dummy scsi_wait_scan module
+ unloading the dummy scsi_wait_scan module
+ Loaded the dummy scsi_wait_scan module
+ unloading the dummy scsi_wait_scan module
  EXT4-fs (dm-0): re-mounted. Opts: (null)
  EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
- SELinux: initialized (dev sda1, type ext4), uses xattr
  EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
- SELinux: initialized (dev dm-2, type ext4), uses xattr
  Adding 3735548k swap on /dev/mapper/vg_hp8300e01-lv_swap.  Priority:-1 extents:1 across:3735548k FS
- SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
  NET: Registered protocol family 10
  e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
  e1000e 0000:00:19.0: irq 29 for MSI/MSI-X
  IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
  e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
  IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
- type=1305 audit(1483703888.868:4): audit_pid=3345 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
- BUG: using smp_processor_id() in preemptible [00000000 00000000] code: beah-beaker-bac/3868
- caller is native_sched_clock+0x49/0x90
- CPU: 0 PID: 3868 Comm: beah-beaker-bac Not tainted 3.10.0-327.rt56.199.el6rt.x86_64.debug #1
- Hardware name: Hewlett-Packard  /3396, BIOS K01 v02.51 06/15/2012
-  ffff880067a57fd8 ffff880067a57ca8 ffffffff81623353 ffff880067a57cc8
-  ffffffff812f4494 00000065af7f4b5e ffff88006455de00 ffff880067a57ce8
-  ffffffff8100cc89 ffff880067a57d57 ffff880068ce0000 ffff880067a57cf8
- Call Trace:

Comment 13 errata-xmlrpc 2017-01-17 18:01:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0113.html