Bug 733406

Summary: 3.1 kernel, scheduling while atomic: kworker/0:1/0/0x00000000 no locks held by kworker/0:1/0
Product: [Fedora] Fedora Reporter: Mark Hamzy <hamzy>
Component: kernelAssignee: Haren Myneni <hbabu>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 16CC: benh, bugproxy, gansalmon, itamar, jonathan, karsten, kernel-maint, madhu.chinakonda
Target Milestone: ---   
Target Release: ---   
Hardware: ppc64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-10-04 14:56:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 718269    
Attachments:
Description Flags
.config file foer kernel 3.1.0-0.rc2.git7.2.fc16.ppc64 none

Description Mark Hamzy 2011-08-25 16:07:51 UTC 
Description of problem:
Booting Fedora-20110825-ppc64-netinst.iso with the following options

boot: linux vnc=1 serial

I see the following:

Welcome to the 64-bit Fedora 20110824 installer!
Hit <TAB> for boot options.
 
 
Welcome to yaboot version 1.3.16 (Red Hat 1.3.16-4.fc16)
Enter "help" to get some basic usage information
boot: linux vnc=1 serial
Please wait, loading kernel...
Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 04380000, size: 20595 Kbytes
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.1.0-0.rc2.git7.2.fc16.ppc64 (mockbuild@ppc-comm02) (gcc version 4.6.1 20110804 (Red Hat 4.6.1-7) (GCC) ) #1 SMP Mon Aug 22 18:15:25 UTC 2011
Max number of cores passed to firmware: 32 (NR_CPUS = 128)
Calling ibm,client-architecture-support... done
command line: ro root=live:CDLABEL=PBOOT vnc=1 serial
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 000000000579d000
alloc_top : 0000000010000000
alloc_top_hi : 0000000010000000
rmo_top : 0000000010000000
ram_top : 0000000010000000
instantiating rtas at 0x000000000ee90000... done
boot cpu hw idx 0
starting cpu hw idx 4... done
starting cpu hw idx 8... done
starting cpu hw idx 12... done
starting cpu hw idx 16... done
starting cpu hw idx 20... done
starting cpu hw idx 24... done
starting cpu hw idx 28... done
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x000000000589e000 -> 0x000000000589f5a6
Device tree struct 0x00000000058a0000 -> 0x00000000058b6000
Calling quiesce...
returning from prom_init
[ 0.000000] Phyp-dump disabled at boot time
[ 0.000000] Using pSeries machine description
[ 0.000000] Using 1TB segments
[ 0.000000] Found initrd at 0xc000000004380000:0xc00000000579cf64
[ 0.000000] bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 32 cpus.
[ 0.000000] CPU maps initialized for 4 threads per core
[ 0.000000] Starting Linux PPC64 #1 SMP Mon Aug 22 18:15:25 UTC 2011
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x1c
[ 0.000000] physicalMemorySize = 0x410000000
[ 0.000000] htab_hash_mask = 0x1fffff
[ 0.000000] -----------------------------------------------------
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.1.0-0.rc2.git7.2.fc16.ppc64 (mockbuild@ppc-comm02) (gcc version 4.6.1 20110804 (Red Hat 4.6.1-7) (GCC) ) #1 SMP Mon Aug 22 18:15:25 UTC 2011
[ 0.000000] [boot]0012 Setup Arch
[ 0.000000] EEH: No capable adapters found
[ 0.000000] PPC64 nvram contains 15360 bytes
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000000 -> 0x00410000
[ 0.000000] Normal empty
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000000 -> 0x00280000
[ 0.000000] 1: 0x00280000 -> 0x00410000
[ 0.000000] [boot]0015 Setup Done
[ 0.000000] PERCPU: Embedded 463 pages/cpu @c000000010000000 s1865856 r0 d30592 u2097152
[ 0.000000] Built 2 zonelists in Node order, mobility grouping on. Total pages: 4201600
[ 0.000000] Policy zone: DMA
[ 0.000000] Kernel command line: ro root=live:CDLABEL=PBOOT vnc=1 serial
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] freeing bootmem node 0
[ 0.000000] freeing bootmem node 1
[ 0.000000] Memory: 16670568k/17039360k available (19476k kernel code, 368792k reserved, 1732k data, 12573k bss, 8888k init)
[ 0.000000] SLUB: Genslabs=15, HWalign=128, Order=0-3, MinObjects=0, CPUs=32, Nodes=256
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] RCU lockdep checking is enabled.
[ 0.000000] NR_IRQS:512 nr_irqs:512 16
[ 0.000000] clocksource: timebase mult[7d0000] shift[22] registered
[ 0.000000] Console: colour dummy device 80x25
[ 0.000000] console [hvc0] enabled, bootconsole disabled
[ 0.000000] console [hvc0] enabled, bootconsole disabled
[ 0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[ 0.000000] ... MAX_LOCKDEP_SUBCLASSES: 8
[ 0.000000] ... MAX_LOCK_DEPTH: 48
[ 0.000000] ... MAX_LOCKDEP_KEYS: 8191
[ 0.000000] ... CLASSHASH_SIZE: 4096
[ 0.000000] ... MAX_LOCKDEP_ENTRIES: 16384
[ 0.000000] ... MAX_LOCKDEP_CHAINS: 32768
[ 0.000000] ... CHAINHASH_SIZE: 16384
[ 0.000000] memory used by lock dependency info: 6367 kB
[ 0.000000] per task-struct memory footprint: 2688 bytes
[ 0.000000] allocated 136314880 bytes of page_cgroup
[ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[ 0.028047] pid_max: default: 32768 minimum: 301
[ 0.028473] Security Framework initialized
[ 0.028485] SELinux: Initializing.
[ 0.034353] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
[ 0.052034] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[ 0.059439] Mount-cache hash table entries: 256
[ 0.070233] Initializing cgroup subsys cpuacct
[ 0.070273] Initializing cgroup subsys memory
[ 0.071746] Initializing cgroup subsys devices
[ 0.071759] Initializing cgroup subsys freezer
[ 0.071767] Initializing cgroup subsys net_cls
[ 0.071776] Initializing cgroup subsys blkio
[ 0.071814] Initializing cgroup subsys perf_event
[ 0.071965] ftrace: allocating 22116 entries in 131 pages
[ 0.075172] POWER7 performance monitor hardware support registered
[ 0.094880] Brought up 32 CPUs
[ 0.096385] Enabling Asymmetric SMT scheduling
[ 0.099211] devtmpfs: initialized
[ 0.107155] atomic64 test passed
[ 0.107604] NET: Registered protocol family 16
[ 0.107684] IBM eBus Device Driver
[ 0.131894] PCI: Probing PCI hardware
[ 0.142027] bio: create slab <bio-0> at 0
[ 0.142984] vgaarb: loaded
[ 0.144014] SCSI subsystem initialized
[ 0.144731] usbcore: registered new interface driver usbfs
[ 0.144832] usbcore: registered new interface driver hub
[ 0.144993] usbcore: registered new device driver usb
[ 0.146174] NetLabel: Initializing
[ 0.146179] NetLabel: domain hash size = 128
[ 0.146184] NetLabel: protocols = UNLABELED CIPSOv4
[ 0.146274] NetLabel: unlabeled traffic allowed by default
[ 0.292643] DMA-API: preallocated 65536 debug entries
[ 0.292667] DMA-API: debugging enabled by kernel config
[ 0.292708] Switching to clocksource timebase
[ 0.293269] Switched to NOHz mode on CPU #1
[ 0.293308] Switched to NOHz mode on CPU #7
[ 0.293313] Switched to NOHz mode on CPU #31
[ 0.293328] Switched to NOHz mode on CPU #14
[ 0.293382] Switched to NOHz mode on CPU #19
[ 0.293395] Switched to NOHz mode on CPU #21
[ 0.293504] Switched to NOHz mode on CPU #26
[ 0.293509] Switched to NOHz mode on CPU #9
[ 0.293531] Switched to NOHz mode on CPU #4
[ 0.293535] Switched to NOHz mode on CPU #28
[ 0.293586] Switched to NOHz mode on CPU #23
[ 0.293589] Switched to NOHz mode on CPU #16
[ 0.293661] Switched to NOHz mode on CPU #3
[ 0.293739] Switched to NOHz mode on CPU #20
[ 0.293771] Switched to NOHz mode on CPU #13
[ 0.293897] Switched to NOHz mode on CPU #6
[ 0.293901] Switched to NOHz mode on CPU #11
[ 0.293915] Switched to NOHz mode on CPU #30
[ 0.293926] Switched to NOHz mode on CPU #18
[ 0.293930] Switched to NOHz mode on CPU #25
[ 0.293964] Switched to NOHz mode on CPU #22
[ 0.293967] Switched to NOHz mode on CPU #15
[ 0.293971] Switched to NOHz mode on CPU #8
[ 0.293983] Switched to NOHz mode on CPU #29
[ 0.294003] Switched to NOHz mode on CPU #27
[ 0.294082] Switched to NOHz mode on CPU #0
[ 0.294085] Switched to NOHz mode on CPU #2
[ 0.294094] Switched to NOHz mode on CPU #12
[ 0.294114] Switched to NOHz mode on CPU #10
[ 0.294125] Switched to NOHz mode on CPU #5
[ 0.294139] Switched to NOHz mode on CPU #17
[ 0.294150] Switched to NOHz mode on CPU #24
[ 0.399418] NET: Registered protocol family 2
[ 0.400486] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
[ 0.404947] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[ 0.409775] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes)
[ 0.416644] TCP: Hash tables configured (established 524288 bind 65536)
[ 0.416684] TCP reno registered
[ 0.417246] UDP hash table entries: 16384 (order: 9, 3145728 bytes)
[ 0.421515] UDP-Lite hash table entries: 16384 (order: 9, 3145728 bytes)
[ 0.426574] NET: Registered protocol family 1
[ 0.426945] Unpacking initramfs...
[ 3.599676] Freeing initrd memory: 20596k freed
[ 3.605863] vio 30000000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[ 3.621055] IOMMU table initialized, virtual merging enabled
[ 3.640854] vio 4000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[ 3.640982] vio 4001: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[ 3.641088] vio 4002: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[ 3.641237] vio 4004: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
[ 3.643057] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.643066] no locks held by kworker/0:1/0.
[ 3.643071] Modules linked in:
[ 3.643077] Call Trace:
[ 3.643085] [c000000277bc79d0] [c000000000015b98] .show_stack+0x94/0x144 (unreliable)
[ 3.643220] [c000000277bc7a90] [c0000000006d870c] .dump_stack+0x24/0x2c
[ 3.643230] [c000000277bc7b10] [c0000000006d9b48] .__schedule_bug+0x90/0x9c
[ 3.643239] [c000000277bc7ba0] [c0000000006ca7ec] .schedule+0xe0/0xb88
[ 3.643248] [c000000277bc7e20] [c000000000017a08] .cpu_idle+0x280/0x2a4
[ 3.643258] [c000000277bc7ec0] [c000000000710ca0] .start_secondary+0x57c/0x584
[ 3.643269] [c000000277bc7f90] [c0000000000092f4] .start_secondary_prolog+0x10/0x14
[ 3.643571] audit: initializing netlink socket (disabled)
[ 3.643636] type=2000 audit(1314286453.639:1): initialized
[ 3.779856] BUG: scheduling while atomic: kworker/0:0/0/0x00000000
[ 3.779866] no locks held by kworker/0:0/0.
[ 3.779871] Modules linked in:
[ 3.779877] Call Trace:
[ 3.779883] [c000000277ad79d0] [c000000000015b98] .show_stack+0x94/0x144 (unreliable)
[ 3.779895] [c000000277ad7a90] [c0000000006d870c] .dump_stack+0x24/0x2c
[ 3.779904] [c000000277ad7b10] [c0000000006d9b48] .__schedule_bug+0x90/0x9c
[ 3.779914] [c000000277ad7ba0] [c0000000006ca7ec] .schedule+0xe0/0xb88
[ 3.779923] [c000000277ad7e20] [c000000000017a08] .cpu_idle+0x280/0x2a4
[ 3.779932] [c000000277ad7ec0] [c000000000710ca0] .start_secondary+0x57c/0x584
[ 3.779942] [c000000277ad7f90] [c0000000000092f4] .start_secondary_prolog+0x10/0x14
[ 3.780089] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780111] no locks held by kworker/0:1/0.
[ 3.780116] Modules linked in:
[ 3.780121] Call Trace:
[ 3.780126] [c000000277bc79d0] [c000000000015b98] .show_stack+0x94/0x144 (unreliable)
[ 3.780137] [c000000277bc7a90] [c0000000006d870c] .dump_stack+0x24/0x2c
[ 3.780146] [c000000277bc7b10] [c0000000006d9b48] .__schedule_bug+0x90/0x9c
[ 3.780155] [c000000277bc7ba0] [c0000000006ca7ec] .schedule+0xe0/0xb88
[ 3.780164] [c000000277bc7e20] [c000000000017a08] .cpu_idle+0x280/0x2a4
[ 3.780174] [c000000277bc7ec0] [c000000000710ca0] .start_secondary+0x57c/0x584
[ 3.780183] [c000000277bc7f90] [c0000000000092f4] .start_secondary_prolog+0x10/0x14
[ 3.780260] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780267] BUG: scheduling while atomic: kworker/0:0/0/0x00000000
[ 3.780269] no locks held by kworker/0:0/0.
[ 3.780272] Modules linked in:
[ 3.780274] Call Trace:
[ 3.780277] [c000000277ad79d0] [c000000000015b98] .show_stack+0x94/0x144 (unreliable)
[ 3.780281] [c000000277ad7a90] [c0000000006d870c] .dump_stack+0x24/0x2c
[ 3.780285] [c000000277ad7b10] [c0000000006d9b48] .__schedule_bug+0x90/0x9c
[ 3.780289] [c000000277ad7ba0] [c0000000006ca7ec] .schedule+0xe0/0xb88
[ 3.780292] [c000000277ad7e20] [c000000000017a08] .cpu_idle+0x280/0x2a4
[ 3.780296] [c000000277ad7ec0] [c000000000710ca0] .start_secondary+0x57c/0x584
[ 3.780300] [c000000277ad7f90] [c0000000000092f4] .start_secondary_prolog+0x10/0x14
[ 3.780317] no locks held by kworker/0:1/0.
[ 3.780322] Modules linked in:
[ 3.780328] Call Trace:
[ 3.780333] [c000000277bc79d0] [c000000000015b98] .show_stack+0x94/0x144 (unreliable)
[ 3.780344] [c000000277bc7a90] [c0000000006d870c] .dump_stack+0x24/0x2c
[ 3.780353] [c000000277bc7b10] [c0000000006d9b48] .__schedule_bug+0x90/0x9c
[ 3.780363] [c000000277bc7ba0] [c0000000006ca7ec] .schedule+0xe0/0xb88
[ 3.780372] [c000000277bc7e20] [c000000000017a08] .cpu_idle+0x280/0x2a4
[ 3.780381] [c000000277bc7ec0] [c000000000710ca0] .start_secondary+0x57c/0x584
[ 3.780385] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780388] no locks held by kworker/0:1/0.
[ 3.780392] Modules linked in:
[ 3.780394] Call Trace:
[ 3.780403] [c000000277ca79d0] [c000000000015b98] .show_stack+0x94/0x144 (unreliable)
[ 3.780411] [c000000277ca7a90] [c0000000006d870c] .dump_stack+0x24/0x2c
[ 3.780417] [c000000277ca7b10] [c0000000006d9b48] .__schedule_bug+0x90/0x9c
[ 3.780425] [c000000277ca7ba0] [c0000000006ca7ec] .schedule+0xe0/0xb88
[ 3.780429] [c000000277ca7e20] [c000000000017a08] .cpu_idle+0x280/0x2a4
[ 3.780433] [c000000277ca7ec0] [c000000000710ca0] .start_secondary+0x57c/0x584
[ 3.780437] [c000000277ca7f90] [c0000000000092f4] .start_secondary_prolog+0x10/0x14
[ 3.780459]
[ 3.780464] [c000000277bc7f90] [c0000000000092f4] .start_secondary_prolog+0x10/0x14
[ 3.780492] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780506] no locks held by kworker/0:1/0.
[ 3.780511] Modules linked in:
[ 3.780517] Call Trace:
[ 3.780522] [c000000277ca79d0] [c000000000015b98] .show_stack+0x94/0x144 (unreliable)
[ 3.780533] [c000000277ca7a90] [c0000000006d870c] .dump_stack+0x24/0x2c
[ 3.780542] [c000000277ca7b10] [c0000000006d9b48] .__schedule_bug+0x90/0x9c
[ 3.780551] [c000000277ca7ba0] [c0000000006ca7ec] .schedule+0xe0/0xb88
[ 3.780560] [c000000277ca7e20] [c000000000017a08] .cpu_idle+0x280/0x2a4
[ 3.780570] [c000000277ca7ec0] [c000000000710ca0] .start_secondary+0x57c/0x584
[ 3.780580] [c000000277ca7f90] [c0000000000092f4] .start_secondary_prolog+0x10/0x14
[ 3.780608] BUG: scheduling while atomic: kworker/0:0/0/0x00000000
[ 3.780614] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780620] no locks held by kworker/0:1/0.
[ 3.780626] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780630] Modules linked in:no locks held by kworker/0:1/0.
[ 3.780638] Modules linked in:
[ 3.780643]
[ 3.780649] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780660] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780671] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780683] no locks held by kworker/0:1/0.
[ 3.780691] Call Trace:
[ 3.780702] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780716] no locks held by kworker/0:1/0.
[ 3.780729] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780738] no locks held by kworker/0:1/0.
[ 3.780747] Modules linked in:
[ 3.780758] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780769] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780779] no locks held by kworker/0:1/0.
[ 3.780790] no locks held by kworker/0:1/0.
[ 3.780800] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780810] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780821] Modules linked in:no locks held by kworker/0:1/0.
[ 3.780839] [c000000277b139d0] [c000000000015b98] .show_stack+0x94/0x144
[ 3.780853] Modules linked in:no locks held by kworker/0:1/0.
[ 3.780872] Modules linked in:
[ 3.780887]
[ 3.780903] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780921] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780932] no locks held by kworker/0:1/0.
[ 3.780945] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780956] Call Trace:
[ 3.780969] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780981] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.780992] (unreliable)
[ 3.781002] Modules linked in:
[ 3.781013] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.781029] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.781044] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.781059] BUG: scheduling while atomic: kworker/0:1/0/0x00000000
[ 3.781073] no locks held by kworker/0:1/0.

This continues on in an infinite loop...
Comment 1 Josh Boyer 2011-08-25 17:31:59 UTC 
Haren, can you take a look at this?
Comment 2 Benjamin Herrenschmidt 2011-08-26 21:55:12 UTC 
The DMA mask warning is harmless, it's just the vio core that is a bit too verbose, it's normal for some vio devices to have no DMA properties associated with them (such as the virtual console for example).

However the scheduling while atomic don't look good, it looks to me like the preempt count isn't properly initialized, it's funny that i never hit that with upstream before though.

Can somebody email me privately the kernel .config used ?
Comment 3 Mark Hamzy 2011-08-26 22:04:36 UTC 
Hey Karsten,

Could you email Ben the kernel .config used for the build, please?
Comment 4 Karsten Hopp 2011-08-29 10:49:07 UTC 
Created attachment 520342 [details]
.config file foer kernel 3.1.0-0.rc2.git7.2.fc16.ppc64
Comment 5 IBM Bug Proxy 2011-09-15 04:00:55 UTC 
------- Comment From anton.com 2011-09-14 23:52 EDT-------
Renaming bug, the iommu warning is harmless and unrelated to this issue.
Comment 6 Karsten Hopp 2011-10-04 14:56:06 UTC 
doesn't happen anymore with the latest kernels, closing