Requesting this be an exception for RHEL6.1.
The patch eliminates for divide-by-zero conditions
that have caused the kernel panic.

Created attachment 481395 [details]
Patch tested with 2.6.32-117


Posted patch.

I will reopen if we see this again --- testing on a 4-rack
machine at the end of the week.

George

*** This bug has been marked as a duplicate of bug 644903 ***

Not seeing this on snap4.

Reopening.

We are seeing this on a system that is about to ship to a customer
with Rhel6.1-ga installed.

I am investigating what might be special about the machine in
manufacturing relative to the test machines.

======================================================


Linux version 2.6.32-131.0.15.el6.x86_64 (mockbuild.bos.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) )
 #1 SMP Tue May 10 15:42:40 EDT 2011


ACPI: Preparing to enter system sleep state S5
Disabling non-boot CPUs ...
Broke affinity for irq 112
Broke affinity for irq 4
Broke affinity for irq 105
Broke affinity for irq 18
Broke affinity for irq 106
Broke affinity for irq 21
Broke affinity for irq 107
Broke affinity for irq 24
Broke affinity for irq 108
Broke affinity for irq 109
Broke affinity for irq 110
Broke affinity for irq 97
Broke affinity for irq 98
Broke affinity for irq 99
Broke affinity for irq 100
Broke affinity for irq 101
Broke affinity for irq 102
Broke affinity for irq 103
Broke affinity for irq 104
divide error: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-4/1-4:1.0/host1/target1:0:0/1:0:0:0/block/sr0/dev
CPU 644 
Modules linked in: autofs4 sunrpc ip6t_REJECT ipv6 vfat fat dm_mirror dm_region_hash dm_log numatools(U) xvma(U) uv_mmtimer(U) hwperf(U) uinput microcode ghes hed ixgbe mdio i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma sg igb dca ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif usb_storage mptsas mptscsih mptbase scsi_transport_sas dm_mod [last unloaded: ip_tables]

Modules linked in: autofs4 sunrpc ip6t_REJECT ipv6 vfat fat dm_mirror dm_region_hash dm_log numatools(U) xvma(U) uv_mmtimer(U) hwperf(U) uinput microcode ghes hed ixgbe mdio i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma sg igb dca ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif usb_storage mptsas mptscsih mptbase scsi_transport_sas dm_mod [last unloaded: ip_tables]
Pid: 1936, comm: migration/644 Tainted: G        W  ----------------   2.6.32-131.0.15.el6.x86_64 #1 Stoutland Platform
RIP: 0010:[<ffffffff81053b65>]  [<ffffffff81053b65>] find_busiest_group+0x5c5/0xb20
RSP: 0018:ffff88087bf0fba0  EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff88087bf0fdbc RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000300 RDI: 0000000000000000
RBP: ffff88087bf0fd30 R08: ffff8a800e450be0 R09: 0000000000000300
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000015f80 R14: ffffffffffffffff R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8a800e500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f2b233c7098 CR3: 0000000001a25000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/644 (pid: 1936, threadinfo ffff88087bf0e000, task ffff88087bf080c0)
Stack:
 ffff88087bf0fcd0 ffff88087bf0fc40 0000000000000000 0000000000000000
<0> 0000000000000000 ffff88087bf0fda8 0000000000000000 0000028400000002
<0> ffffffff00000000 ffff8a800e5108c0 0000000000000000 0000000000000008
Call Trace:
 [<ffffffff814db693>] thread_return+0x3aa/0x777
 [<ffffffff8100bc8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff810c3690>] ? stop_machine_cpu_stop+0x0/0xe0
 [<ffffffff810c3605>] cpu_stopper_thread+0x125/0x1b0
 [<ffffffff814db337>] ? thread_return+0x4e/0x777
 [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
 [<ffffffff810c34e0>] ? cpu_stopper_thread+0x0/0x1b0
 [<ffffffff8108ddf6>] kthread+0x96/0xa0
 [<ffffffff8100c1ca>] child_rip+0xa/0x20
 [<ffffffff8108dd60>] ? kthread+0x0/0xa0
 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20

In the case I saw, the customer was running linux-2.6.32-71.7.1.el6.x86_64.  As is indicated, that might be fixed by 644903.

Yes, that caught another problem, but
the one I reopened for is in 6.1.

George

We're also seeing this bug in with KVM on RHEL 6.1 and a RHEL 6.1 guest on shutdown.



555 Halting system...
556 md: stopping all md devices.
557 ACPI: Preparing to enter system sleep state S5
558 Disabling non-boot CPUs ...
559 BUG: soft lockup - CPU#0 stuck for 67s! [migration/0:5]
560 Modules linked in: nfs lockd fscache(T) nfs_acl auth_rpcgss autofs4 sunrpc ipv6 dm_mirror dm_region_hash dm_log microcode virtio_balloon virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_    generic ata_piix dm_mod [last unloaded: speedstep_lib]
561 CPU 0:
562 Modules linked in: nfs lockd fscache(T) nfs_acl auth_rpcgss autofs4 sunrpc ipv6 dm_mirror dm_region_hash dm_log microcode virtio_balloon virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_    generic ata_piix dm_mod [last unloaded: speedstep_lib]
563 Pid: 5, comm: migration/0 Tainted: G           ---------------- T 2.6.32-131.0.15.el6.x86_64 #1 KVM
564 RIP: 0010:[<ffffffff810c36ff>]  [<ffffffff810c36ff>] stop_machine_cpu_stop+0x6f/0xe0
565 RSP: 0018:ffff880198dabdd0  EFLAGS: 00000293
566 RAX: 0000000000000001 RBX: ffff880198dabdf0 RCX: ffff8800282111e8
567 RDX: 0000000000000000 RSI: ffff88019752c040 RDI: ffff880197161d28
568 RBP: ffffffff8100bc8e R08: ffff880198daa000 R09: 0000000000000001
569 R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000000
570 R13: ffffffff814db337 R14: ffff880198dabdf0 R15: ffff8801936e8f00
571 FS:  0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
572 CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
573 CR2: 00007fff295baeb0 CR3: 000000019362b000 CR4: 00000000000006f0
574 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
575 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
576 Call Trace:
577  [<ffffffff810c3690>] ? stop_machine_cpu_stop+0x0/0xe0
578  [<ffffffff810c35ba>] ? cpu_stopper_thread+0xda/0x1b0
579  [<ffffffff814db337>] ? thread_return+0x4e/0x777
580  [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
581  [<ffffffff810c34e0>] ? cpu_stopper_thread+0x0/0x1b0
582  [<ffffffff8108ddf6>] ? kthread+0x96/0xa0
583  [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
584  [<ffffffff8108dd60>] ? kthread+0x0/0xa0
585  [<ffffffff8100c1c0>] ? child_rip+0x0/0x20

I have not seen this on rhel6.3 in all of our testing.
Is anyone else still seeing this?  Jason?

George

George, I found the actual cause of this and fixed in RHEL6.3.  Sorry I didnt find and close this BZ as a DUP of BZ785959, I think you should do that now...


>>>email sent to rhkernel-list by me on 02/23/2012 05:51 PM

[RHEL6.3 Patch] Fix Kernel divide by zero panic in find_busiest_group()
--------------------------------------------------------------------------------
RHEL6 is missing the attached upstream patch that at first glance appears to
be just a cleanup.  However this patch changes init_sched_groups_power()
to call update_group_power() which in turn calls update_cpu_power().  The
update_cpu_power() function verifies that the sched_group->cpu_power
is not initialized with a zero where init_sched_groups_power() did not do this.
Since find_busiest_group() uses sched_group->cpu_power in the demoninator
of a fraction it must not be zero!

-----------------------------------------------------------------------------------------------------
static void update_cpu_power(struct sched_domain *sd, int cpu)
{
...
        if (sched_feat(ARCH_POWER))
                power *= arch_scale_freq_power(sd, cpu);
        else
                power *= default_scale_freq_power(sd, cpu);

        power >>= SCHED_LOAD_SHIFT;

        power *= scale_rt_power(cpu);
        power >>= SCHED_LOAD_SHIFT;

>>>if (!power)
>>>        power = 1;

        sdg->cpu_power = power;
}
-------------------------------------------------------------------------------------------------------

The attached patch fixes this panic and BZ785959.  Since I can not reproduce
this problem I'm building a kernel in brew for the customers to test and verify
the fix.  I'll respond with the results once they verify it.



rhel6-cpu_power_init.patch

commit d274cb30f4a08045492d3f0c47cdf1a25668b1f5
Author: Peter Zijlstra <a.p.zijlstra>
Date:   Thu Apr 7 14:09:43 2011 +0200

    sched: Simplify ->cpu_power initialization
    
    The code in update_group_power() does what init_sched_groups_power()
    does and more, so remove the special init_ code and call the generic
    code instead.
    
    Also move the sd->span_weight initialization because
    update_group_power() needs it.
    
    Signed-off-by: Peter Zijlstra <a.p.zijlstra>
    Cc: Mike Galbraith <efault>
    Cc: Nick Piggin <npiggin>
    Cc: Linus Torvalds <torvalds>
    Cc: Andrew Morton <akpm>
    Link: http://lkml.kernel.org/r/20110407122941.875856012@chello.nl
    Signed-off-by: Ingo Molnar <mingo>

diff --git a/kernel/sched.c b/kernel/sched.c
index 3ce2ab6..071cf49 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -8618,9 +8618,6 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)
 	struct rq *rq = cpu_rq(cpu);
 	struct sched_domain *tmp;
 
-	for (tmp = sd; tmp; tmp = tmp->parent)
-		tmp->span_weight = cpumask_weight(sched_domain_span(tmp));
-
 	/* Remove the sched domains which do not contribute to scheduling. */
 	for (tmp = sd; tmp; ) {
 		struct sched_domain *parent = tmp->parent;
@@ -9098,46 +9095,12 @@ static void free_sched_groups(const struct cpumask *cpu_map,
  */
 static void init_sched_groups_power(int cpu, struct sched_domain *sd)
 {
-	struct sched_domain *child;
-	struct sched_group *group;
-	long power;
-	int weight;
-
 	WARN_ON(!sd || !sd->groups);
 
 	if (cpu != group_first_cpu(sd->groups))
 		return;
 
-	child = sd->child;
-
-	sd->groups->cpu_power = 0;
-
-	if (!child) {
-		power = SCHED_LOAD_SCALE;
-		weight = cpumask_weight(sched_domain_span(sd));
-		/*
-		 * SMT siblings share the power of a single core.
-		 * Usually multiple threads get a better yield out of
-		 * that one core than a single thread would have,
-		 * reflect that in sd->smt_gain.
-		 */
-		if ((sd->flags & SD_SHARE_CPUPOWER) && weight > 1) {
-			power *= sd->smt_gain;
-			power /= weight;
-			power >>= SCHED_LOAD_SHIFT;
-		}
-		sd->groups->cpu_power += power;
-		return;
-	}
-
-	/*
-	 * Add cpu_power of each child group to this groups cpu_power.
-	 */
-	group = child->groups;
-	do {
-		sd->groups->cpu_power += group->cpu_power;
-		group = group->next;
-	} while (group != child->groups);
+	update_group_power(sd, cpu);
 }
 
 /*
@@ -9444,7 +9407,7 @@ static int __build_sched_domains(const struct cpumask *cpu_map,
 {
 	enum s_alloc alloc_state = sa_none;
 	struct s_data d;
-	struct sched_domain *sd;
+	struct sched_domain *sd, *tmp;
 	int i;
 #ifdef CONFIG_NUMA
 	d.sd_allnodes = 0;
@@ -9467,6 +9430,9 @@ static int __build_sched_domains(const struct cpumask *cpu_map,
 		sd = __build_book_sched_domain(&d, cpu_map, attr, sd, i);
 		sd = __build_mc_sched_domain(&d, cpu_map, attr, sd, i);
 		sd = __build_smt_sched_domain(&d, cpu_map, attr, sd, i);
+
+		for (tmp = sd; tmp; tmp = tmp->parent)
+			tmp->span_weight = cpumask_weight(sched_domain_span(tmp));
 	}
 
 	for_each_cpu(i, cpu_map) {

John or Larry,

I can't access 784959 so I can't close this as a duplicate.

I agree it should be closed.

*** This bug has been marked as a duplicate of bug 784959 ***