Created attachment 824906 [details]
File: dmesg

Was this after a resume from suspend/hibernate or did it happen randomly?

*** Bug 1025045 has been marked as a duplicate of this bug. ***

Just got this crash again this morning during an Apper update.

Description of problem:
This showed up upon resume from suspend.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.1.9
cmdline:        BOOT_IMAGE=/vmlinuz-3.11.10-300.fc20.x86_64 root=UUID=c07db0f8-16c8-4d06-9f59-5a04a34fb2b1 ro rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8
kernel:         3.11.10-300.fc20.x86_64
runlevel:       N 5
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 1 PID: 0 at kernel/time/tick-broadcast.c:667 tick_broadcast_oneshot_control+0x170/0x180()
Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle tun bridge stp llc ebtable_nat ebtables fuse arc4 ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack lm90 nf_conntrack ip6_tables snd_hda_codec_analog b43 bcma snd_hda_intel mac80211 snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm cfg80211 powernow_k8 kvm_amd snd_page_alloc kvm snd_timer snd tg3 ssb hp_wmi ptp pps_core sparse_keymap rfkill soundcore edac_core ppdev k8temp edac_mce_amd mmc_core serio_raw sp5100_tco hp_accel i2c_piix4 shpchp wmi parport_pc video lis3lv02d input_polldev parport mperf binfmt_misc radeon firewire_ohci i2c_algo_bit ata_generic pata_acpi firewire_core drm_kms_helper crc_itu_t yenta_socket ttm pata_atiixp drm i2c_core
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.11.10-300.fc20.x86_64 #1
Hardware name: Hewlett-Packard HP Compaq 6715b (RK156AV)/30C2, BIOS 68YTT Ver. F.07 07/16/2007
 0000000000000009 ffff880139ea7de8 ffffffff816441bb 0000000000000000
 ffff880139ea7e20 ffffffff8106715d 0000000000000001 0000000000000004
 ffff88013fd0dc80 0000000000000002 ffffffff81cb7400 ffff880139ea7e30
Call Trace:
 [<ffffffff816441bb>] dump_stack+0x45/0x56
 [<ffffffff8106715d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff8106723a>] warn_slowpath_null+0x1a/0x20
 [<ffffffff810c02d0>] tick_broadcast_oneshot_control+0x170/0x180
 [<ffffffff810beb38>] clockevents_notify+0x178/0x1a0
 [<ffffffff8101ae95>] amd_e400_idle+0x75/0x100
 [<ffffffff8101b646>] arch_cpu_idle+0x26/0x30
 [<ffffffff810b6695>] cpu_startup_entry+0xe5/0x280
 [<ffffffff8103ed47>] start_secondary+0x217/0x2c0

Description of problem:
Turned laptop on - from 'Power Off' state. Logged in to Mate, when desktop loaded, bugzilla reported error.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.1.9
cmdline:        BOOT_IMAGE=/vmlinuz-3.11.9-200.fc19.x86_64 root=/dev/mapper/fedora_satellite-root ro rd.md=0 rd.lvm.lv=fedora_satellite/swap rd.dm=0 rd.lvm.lv=fedora_satellite/root vconsole.keymap=us rd.luks=0 vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=en_US.UTF-8
kernel:         3.11.9-200.fc19.x86_64
runlevel:       unknown
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 0 PID: 0 at kernel/time/tick-broadcast.c:667 tick_broadcast_oneshot_control+0x170/0x180()
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.11.9-200.fc19.x86_64 #1
Hardware name: TOSHIBA Satellite A215/IALAA, BIOS V2.00 06/10/2008
 0000000000000009 ffffffff81c01e00 ffffffff8164764b 0000000000000000
 ffffffff81c01e38 ffffffff8106715d 0000000000000000 0000000000000004
 ffff8800b7a0dc80 0000000000000002 ffffffff81c238c0 ffffffff81c01e48
Call Trace:
 [<ffffffff8164764b>] dump_stack+0x45/0x56
 [<ffffffff8106715d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff8106723a>] warn_slowpath_null+0x1a/0x20
 [<ffffffff810c02f0>] tick_broadcast_oneshot_control+0x170/0x180
 [<ffffffff810beb58>] clockevents_notify+0x178/0x1a0
 [<ffffffff8101ae65>] amd_e400_idle+0x75/0x100
 [<ffffffff8101b616>] arch_cpu_idle+0x26/0x30
 [<ffffffff810b669e>] cpu_startup_entry+0xce/0x280
 [<ffffffff8163d937>] rest_init+0x77/0x80
 [<ffffffff81d0eef5>] start_kernel+0x40a/0x416
 [<ffffffff81d0e8f4>] ? repair_env_string+0x5c/0x5c
 [<ffffffff81d0e120>] ? early_idt_handlers+0x120/0x120
 [<ffffffff81d0e5dc>] x86_64_start_reservations+0x2a/0x2c
 [<ffffffff81d0e6e6>] x86_64_start_kernel+0x108/0x117

Description of problem:
Happened immediately after startup.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.1.9
cmdline:        BOOT_IMAGE=/boot/vmlinuz-3.11.9-200.fc19.x86_64 root=UUID=219f2749-f79a-4770-9aa3-7d70e25ae21a ro rd.md=0 rd.lvm=0 rd.dm=0 vconsole.keymap=us rd.luks=0 vconsole.font=latarcyrheb-sun16 LANG=en_US.UTF-8 rhgb quiet
kernel:         3.11.9-200.fc19.x86_64
runlevel:       unknown
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 0 PID: 0 at kernel/time/tick-broadcast.c:667 tick_broadcast_oneshot_control+0x170/0x180()
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.11.9-200.fc19.x86_64 #1
Hardware name: Micro-Star International MS-1244/MS-1244, BIOS A1244AMS Ver1.0B 09/30/2010
 0000000000000009 ffffffff81c01e00 ffffffff8164764b 0000000000000000
 ffffffff81c01e38 ffffffff8106715d 0000000000000000 0000000000000004
 ffff88022fc0dc80 0000000000000002 ffffffff81c238c0 ffffffff81c01e48
Call Trace:
 [<ffffffff8164764b>] dump_stack+0x45/0x56
 [<ffffffff8106715d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff8106723a>] warn_slowpath_null+0x1a/0x20
 [<ffffffff810c02f0>] tick_broadcast_oneshot_control+0x170/0x180
 [<ffffffff810beb58>] clockevents_notify+0x178/0x1a0
 [<ffffffff8101ae65>] amd_e400_idle+0x75/0x100
 [<ffffffff8101b616>] arch_cpu_idle+0x26/0x30
 [<ffffffff810b669e>] cpu_startup_entry+0xce/0x280
 [<ffffffff8163d937>] rest_init+0x77/0x80
 [<ffffffff81d0eef5>] start_kernel+0x40a/0x416
 [<ffffffff81d0e8f4>] ? repair_env_string+0x5c/0x5c
 [<ffffffff81d0e120>] ? early_idt_handlers+0x120/0x120
 [<ffffffff81d0e5dc>] x86_64_start_reservations+0x2a/0x2c
 [<ffffffff81d0e6e6>] x86_64_start_kernel+0x108/0x117

Description of problem:
Happened immediately after startup.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.1.9
cmdline:        BOOT_IMAGE=/boot/vmlinuz-3.11.10-200.fc19.x86_64 root=UUID=219f2749-f79a-4770-9aa3-7d70e25ae21a ro rd.md=0 rd.lvm=0 rd.dm=0 vconsole.keymap=us rd.luks=0 vconsole.font=latarcyrheb-sun16 LANG=en_US.UTF-8 rhgb quiet
kernel:         3.11.10-200.fc19.x86_64
runlevel:       unknown
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 0 PID: 0 at kernel/time/tick-broadcast.c:667 tick_broadcast_oneshot_control+0x170/0x180()
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.11.10-200.fc19.x86_64 #1
Hardware name: Micro-Star International MS-1244/MS-1244, BIOS A1244AMS Ver1.0B 09/30/2010
 0000000000000009 ffffffff81c01df8 ffffffff8164768b 0000000000000000
 ffffffff81c01e30 ffffffff8106715d 0000000000000000 0000000000000004
 ffff88022fc0dc80 0000000000000002 ffffffff81c238c0 ffffffff81c01e40
Call Trace:
 [<ffffffff8164768b>] dump_stack+0x45/0x56
 [<ffffffff8106715d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff8106723a>] warn_slowpath_null+0x1a/0x20
 [<ffffffff810c02f0>] tick_broadcast_oneshot_control+0x170/0x180
 [<ffffffff810beb58>] clockevents_notify+0x178/0x1a0
 [<ffffffff8101ae65>] amd_e400_idle+0x75/0x100
 [<ffffffff8101b616>] arch_cpu_idle+0x26/0x30
 [<ffffffff810b66b5>] cpu_startup_entry+0xe5/0x280
 [<ffffffff8163d977>] rest_init+0x77/0x80
 [<ffffffff81d0eef5>] start_kernel+0x40a/0x416
 [<ffffffff81d0e8f4>] ? repair_env_string+0x5c/0x5c
 [<ffffffff81d0e120>] ? early_idt_handlers+0x120/0x120
 [<ffffffff81d0e5dc>] x86_64_start_reservations+0x2a/0x2c
 [<ffffffff81d0e6e6>] x86_64_start_kernel+0x108/0x117

In both:
https://bugzilla.redhat.com/show_bug.cgi?id=989718
https://bugzilla.redhat.com/show_bug.cgi?id=996973

reporters mentioned that sometime in 3.11.X the issue had gone.
So this one is just not fixed yet.

Issue goes back in similar forms to 3.1.X:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=656196

This issue still occurs regularly on my machine.

(In reply to Fred Erickson from comment #6)
> Description of problem:
> Turned laptop on - from 'Power Off' state. Logged in to Mate, when desktop
> loaded, bugzilla reported error.
> 
> Version-Release number of selected component:
> kernel
> 
> Additional info:
> reporter:       libreport-2.1.9
> cmdline:        BOOT_IMAGE=/vmlinuz-3.11.9-200.fc19.x86_64
> root=/dev/mapper/fedora_satellite-root ro rd.md=0
> rd.lvm.lv=fedora_satellite/swap rd.dm=0 rd.lvm.lv=fedora_satellite/root
> vconsole.keymap=us rd.luks=0 vconsole.font=latarcyrheb-sun16 rhgb quiet
> LANG=en_US.UTF-8
> kernel:         3.11.9-200.fc19.x86_64
> runlevel:       unknown
> type:           Kerneloops
> 
> Truncated backtrace:
> WARNING: CPU: 0 PID: 0 at kernel/time/tick-broadcast.c:667
> tick_broadcast_oneshot_control+0x170/0x180()
> Modules linked in:
> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.11.9-200.fc19.x86_64 #1
> Hardware name: TOSHIBA Satellite A215/IALAA, BIOS V2.00 06/10/2008
>  0000000000000009 ffffffff81c01e00 ffffffff8164764b 0000000000000000
>  ffffffff81c01e38 ffffffff8106715d 0000000000000000 0000000000000004
>  ffff8800b7a0dc80 0000000000000002 ffffffff81c238c0 ffffffff81c01e48
> Call Trace:
>  [<ffffffff8164764b>] dump_stack+0x45/0x56
>  [<ffffffff8106715d>] warn_slowpath_common+0x7d/0xa0
>  [<ffffffff8106723a>] warn_slowpath_null+0x1a/0x20
>  [<ffffffff810c02f0>] tick_broadcast_oneshot_control+0x170/0x180
>  [<ffffffff810beb58>] clockevents_notify+0x178/0x1a0
>  [<ffffffff8101ae65>] amd_e400_idle+0x75/0x100
>  [<ffffffff8101b616>] arch_cpu_idle+0x26/0x30
>  [<ffffffff810b669e>] cpu_startup_entry+0xce/0x280
>  [<ffffffff8163d937>] rest_init+0x77/0x80
>  [<ffffffff81d0eef5>] start_kernel+0x40a/0x416
>  [<ffffffff81d0e8f4>] ? repair_env_string+0x5c/0x5c
>  [<ffffffff81d0e120>] ? early_idt_handlers+0x120/0x120
>  [<ffffffff81d0e5dc>] x86_64_start_reservations+0x2a/0x2c
>  [<ffffffff81d0e6e6>] x86_64_start_kernel+0x108/0x117

Additional comments I probably should have added to original: Since F19 was installed on this laptop, the login screen has not been readable. Looks like the screen has been sliced in to a series of irregular triangles like a puzzle and not assembled correctly. I am the only user, so pressing enter, then typing in my password brings up the desktop. 
If I log out later, the login screen is displayed properly. 
I have used Fedora on this laptop since it was new and 19 is the only version I've used that has behaved this way.

(In reply to Michael Convey from comment #10)
> This issue still occurs regularly on my machine.

Does it also happen on kernel 3.12 ? I do not see reports with that kernel version, looks like update fix the problem.

(In reply to Stanislaw Gruszka from comment #12)
> (In reply to Michael Convey from comment #10)
> > This issue still occurs regularly on my machine.
> 
> Does it also happen on kernel 3.12 ? I do not see reports with that kernel
> version, looks like update fix the problem.

Just happened again:
$ uname -rvs
Linux 3.12.7-200.fc19.x86_64 #1 SMP Fri Jan 10 15:32:06 UTC 2014

And again: 
$ uname -rvs
Linux 3.12.8-200.fc19.x86_64 #1 SMP Thu Jan 16 04:18:11 UTC 2014

Created attachment 857157 [details]
amd_e400_c1e_clockevents.patch

Proposed fix for the bug.

I launched kernel build with it here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=6468631
(bz1058268 number in the name is wrong, but that kernel includes proper patch)

Please test it, it can make problem gone, but either can crash the kernel :-)

Any test results /re kernel from comment 15 ?

(In reply to Stanislaw Gruszka from comment #16)
> Any test results /re kernel from comment 15 ?

Forgive my ignorance, can you provide the steps for me to update my kernel with your build? Then, I will run your kernel for a week or so and see if this bug occurs. Will that suffice?

(In reply to Stanislaw Gruszka from comment #16)
> Any test results /re kernel from comment 15 ?

Forgive my ignorance, can you provide the steps for me to update my kernel with your build? Then, I will run your kernel for a week or so and see if this bug occurs. Will that suffice?

You have to download kernel package from above link for your architecture, i.e. for x86_64 it would be:
> http://kojipkgs.fedoraproject.org//work/tasks/8635/6468635/kernel-3.12.9-301.bz1058268.fc20.x86_64.rpm
Then install it using rpm -ivh kernel-3.12.9-301.bz1058268.fc20.x86_64.rpm command. If there will be dependency or other installation problem, you can use --force and --nodeps rpm options. Then boot into new installed kernel.

Sorry for the delay. I just installed your kernel. 
$ uname -rvs
Linux 3.12.9-301.bz1058268.fc20.x86_64 #1 SMP Wed Jan 29 17:26:47 UTC 2014
I'll report back in a week or so (or earlier if this bug shows up).

I just got this bug using the following kernel:

$ uname -rvs
Linux 3.12.9-301.bz1058268.fc20.x86_64 #1 SMP Wed Jan 29 17:26:47 UTC 2014

Created attachment 861273 [details]
tick_broadcast_v2.patch

New patch, hopefully this one would be better.
Kernel build with the launched here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=6511401
Please test it.

*** Bug 1063108 has been marked as a duplicate of this bug. ***

I almost forgot how the kernel ring buffer looks without errors ...
All tests[1] have PASSED. :)
Patched & tested with 3.14.0-0.rc1.git5.7.fc21.1.x86_64.

Thanks Stanislaw!

https://lists.fedoraproject.org/pipermail/users/2014-January/446267.html

Created attachment 861642 [details]
dmesg 3.12.10-300.bz1031296_v2.fc20.x86_64+debug - devices pm_test - S3

For comparison with a 3.14, this[1] happens exclusively with a 3.12.10-300.bz1031296_v2.fc20.x86_64+debug.
But yeah, "tick_broadcast" is gone. :)

[1] https://bugzilla.redhat.com/attachment.cgi?id=861642

(In reply to poma from comment #26)
> For comparison with a 3.14, this[1] happens exclusively with a
> 3.12.10-300.bz1031296_v2.fc20.x86_64+debug.

Not sure how those ATA errors are releted with the patch, they do not happen on 3.12 kernel without the patch? BTW, what is +debug ?

(In reply to Stanislaw Gruszka from comment #27)
> (In reply to poma from comment #26)
> > For comparison with a 3.14, this[1] happens exclusively with a
> > 3.12.10-300.bz1031296_v2.fc20.x86_64+debug.
> 
> Not sure how those ATA errors are releted with the patch, they do not happen
> on 3.12 kernel without the patch? BTW, what is +debug ?

Sorry, It should be stated "FYI" i.e. "FTR".
It's only for comparison(3.14 vs. 3.12) with respect to testing.
Not related to the patch per se.

BTW patched & tested with 3.14.0-0.rc2.git0.10.fc21.x86_64.
"tick_broadcast" is gone.
Everything is OK. ;)

"+debug" supplements kernel's name, 
i.e. /boot/vmlinuz-3.12.10-300.bz1031296_v2.fc20.x86_64+debug

Comment on attachment 861642 [details]
dmesg 3.12.10-300.bz1031296_v2.fc20.x86_64+debug - devices pm_test - S3

It's only for comparison(3.14 vs. 3.12) with respect to testing.
Not related to the patch per se.

http://marc.info/?l=linux-pm&m=139212870024923&q=raw
http://marc.info/?l=linux-kernel&m=139212871024925&q=raw

Subject: tick: Clear broadcast pending bit when switching to oneshot
From: Thomas Gleixner <tglx>
Date: Tue, 11 Feb 2014 14:35:40 +0100

AMD systems which use the C1E workaround in the amd_e400_idle routine
trigger the WARN_ON_ONCE in the broadcast code when onlining a CPU.

The reason is that the idle routine of those AMD systems switches the
cpu into forced broadcast mode early on before the newly brought up
CPU can switch over to high resolution / NOHZ mode. The timer related
CPU1 bringup looks like this:

  clockevent_register_device(local_apic);
  tick_setup(local_apic);
  ...
  idle()
	tick_broadcast_on_off(FORCE);
	tick_broadcast_oneshot_control(ENTER)
	  cpumask_set(cpu, broadcast_oneshot_mask);
	halt();

Now the broadcast interrupt on CPU0 sets CPU1 in the
broadcast_pending_mask and wakes CPU1. So CPU1 continues:

	local_apic_timer_interrupt()
	   tick_handle_periodic();
	   softirq()
	     tick_init_highres();
	       cpumask_clr(cpu, broadcast_oneshot_mask);
	
	tick_broadcast_oneshot_control(ENTER)
	   WARN_ON(cpumask_test(cpu, broadcast_pending_mask);

So while we remove CPU1 from the broadcast_oneshot_mask when we switch
over to highres mode, we do not clear the pending bit, which then
triggers the warning when we go back to idle.

The reason why this is only visible on C1E affected AMD systems is
that the other machines enter the deep sleep states via
acpi_idle/intel_idle and exit the broadcast mode before executing the
remote triggered local_apic_timer_interrupt. So the pending bit is
already cleared when the switch over to highres mode is clearing the
oneshot mask.

The solution is simple: Clear the pending bit together with the mask
bit when we switch over to highres mode.

Reported-by: poma <pomidorabelisima>
Cc: stable.org # 3.10+
Signed-off-by: Thomas Gleixner <tglx>
---
 kernel/time/tick-broadcast.c |    1 +
 1 file changed, 1 insertion(+)

Index: linux-2.6/kernel/time/tick-broadcast.c
===================================================================
--- linux-2.6.orig/kernel/time/tick-broadcast.c
+++ linux-2.6/kernel/time/tick-broadcast.c
@@ -756,6 +756,7 @@ out:
 static void tick_broadcast_clear_oneshot(int cpu)
 {
 	cpumask_clear_cpu(cpu, tick_broadcast_oneshot_mask);
+	cpumask_clear_cpu(cpu, tick_broadcast_pending_mask);
 }
 
 static void tick_broadcast_init_next_event(struct cpumask *mask,

Josh, plese apply above patch as fix for this bug.

Fixed in Fedora git.  Thanks Stanislaw and poma!

kernel-3.13.3-200.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.13.3-200.fc20

kernel-3.12.11-200.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/kernel-3.12.11-200.fc19

Package kernel-3.12.11-200.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.12.11-200.fc19'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-2556/kernel-3.12.11-200.fc19
then log in and leave karma (feedback).

kernel-3.13.3-201.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.