One of our customers experiences reproducible OOM killer actions on his backup server. OOM triggers when the node runs Parallels Virtuozzo Containers kernels based on Red Hat 2.6.18-164.2.1.el5 and 2.6.18-164.10.1.el5 kernels (magaraid_sas driver has version 00.00.04.08-RH2). OOM does not trigger when the node runs PVC kernel based on Red Hat 2.6.18-128.2.1.el5 kernel (magaraid_sas driver has version 00.00.04.01-RH1). He also tried take a new kernel (based on 2.6.18-164.10.1.el5, which is affected by the issue) and compile older megaraid_sas module from sources for it (v00.00.03.24) - works without problems. One more note: when running on affected kernels (i'd say on kernels with affected megaraid_sas driver), logs are filled with the following messages: Dec 14 19:13:40 host kernel: megasas_aen_polling[1]: scanning ... Dec 14 19:13:42 host kernel: Vendor: ATA Model: ST31000340NS Rev: MA08 Dec 14 19:13:45 host kernel: Type: Direct-Access ANSI SCSI revision: 05 Dec 14 19:13:45 host kernel: Vendor: ATA Model: ST31000340NS Rev: MA08 Dec 14 19:13:48 host kernel: Type: Direct-Access ANSI SCSI revision: 05 Dec 14 19:13:54 host kernel: Vendor: ATA Model: ST31000340NS Rev: MA08 Dec 14 19:13:56 host kernel: Type: Direct-Access ANSI SCSI revision: 05 Dec 14 19:13:58 host kernel: Vendor: ATA Model: ST31000340NS Rev: MA08 Dec 14 19:14:00 host kernel: Type: Direct-Access ANSI SCSI revision: 05 ------------------- * Why OOM triggers? Because there is a real memory shortage. * Who allocated all the memory? Alt+SysRQ=M showed that A LOT of memory was used for slab "size-128": --- size-128 : size 3974189056 objsize 128 --- We've built a debug kernel (CONFIG_DEBUG_SLAB=y, CONFIG_DEBUG_SLAB_LEAK=y) and /proc/slab_allocators file showed that there are abnormally high number of "size-128" objects allocated and not freed in megaraid_sas driver: --- size-128: 24236892 megasas_complete_cmd_dpc+0x2a6/0x43d [megaraid_sas] --- This code in question effectively is: megasas_service_aen()->kzalloc(). One more note: the problem occurs on a server with the following controller: # lspci -s 0e:00.0 -xxx -vvv 0e:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04) Subsystem: Dell PERC 6/E Adapter RAID Controller Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 169 Region 0: Memory at fc480000 (64-bit, non-prefetchable) [size=256K] Region 2: I/O ports at ec00 [size=256] Region 3: Memory at fc440000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at fc300000 [disabled] [size=32K] Capabilities: [b0] Express Endpoint IRQ 0 Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag- Device: Latency L0s unlimited, L1 unlimited Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable- Non-Fatal- Fatal+ Unsupported- Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ Device: MaxPayload 256 bytes, MaxReadReq 2048 bytes Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0 Link: Latency L0s <2us, L1 unlimited Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch- Link: Speed 2.5Gb/s, Width x8 Capabilities: [c4] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable- Address: 0000000000000000 Data: 0000 Capabilities: [d4] MSI-X: Enable- Mask- TabSize=4 Vector table: BAR=0 offset=0003e000 PBA: BAR=0 offset=00fff000 Capabilities: [e0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- Capabilities: [ec] Vital Product Data Capabilities: [100] Power Budgeting 00: 00 10 60 00 47 01 18 00 04 00 04 01 10 00 00 00 10: 04 00 48 fc 00 00 00 00 01 ec 00 00 04 00 44 fc 20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 0a 1f 30: 00 00 30 fc b0 00 00 00 00 00 00 00 05 01 00 00 40: 00 00 00 00 09 00 00 00 01 00 00 f0 00 00 00 c0 50: 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 05 00 fc ff ff ff ff ff 00 00 00 20 01 00 00 00 80: 01 ff ff ff 00 00 00 00 00 00 00 00 01 00 fc ff 90: ff ff ff ff 00 00 02 00 08 00 00 00 01 80 ff ff a0: 04 00 00 00 00 00 00 00 00 06 d8 05 70 56 a4 00 b0: 10 c4 01 00 c1 0f 64 00 34 48 08 00 81 d4 03 00 c0: 00 00 81 00 05 d4 84 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 11 e0 03 00 00 e0 03 00 00 f0 ff 00 e0: 01 ec 02 02 00 00 00 00 00 00 0e 00 03 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 The problem DOES NOT occur even with the recent megaraid_sas driver on a node with controller: # lspci -s 03:00.0 -xxx -vvv 03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04) Subsystem: Dell PERC 6/i Integrated RAID Controller Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 90 Region 0: Memory at df180000 (64-bit, non-prefetchable) [size=256K] Region 2: I/O ports at fc00 [size=256] Region 3: Memory at df1c0000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at df100000 [disabled] [size=32K] Capabilities: [b0] Express Endpoint IRQ 0 Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag- Device: Latency L0s unlimited, L1 unlimited Device: AtnBtn- AtnInd- PwrInd- Device: Errors: Correctable- Non-Fatal- Fatal+ Unsupported- Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ Device: MaxPayload 256 bytes, MaxReadReq 2048 bytes Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0 Link: Latency L0s <2us, L1 unlimited Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch- Link: Speed 2.5Gb/s, Width x4 Capabilities: [c4] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable- Address: 0000000000000000 Data: 0000 Capabilities: [d4] MSI-X: Enable- Mask- TabSize=4 Vector table: BAR=0 offset=0003e000 PBA: BAR=0 offset=00fff000 Capabilities: [e0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- Capabilities: [ec] Vital Product Data Capabilities: [100] Power Budgeting 00: 00 10 60 00 07 00 10 00 04 00 04 01 10 00 00 00 10: 04 00 18 df 00 00 00 00 01 fc 00 00 04 00 1c df 20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 0c 1f 30: 00 00 10 df b0 00 00 00 00 00 00 00 0f 01 00 00 40: 00 00 00 00 09 00 00 00 01 00 00 f0 00 00 00 c0 50: 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 05 00 fc ff ff ff ff ff 00 00 00 20 01 00 00 00 80: 01 ff ff ff 00 00 00 00 00 00 00 00 01 00 fc ff 90: ff ff ff ff 00 00 02 00 08 00 00 00 01 80 ff ff a0: 04 00 00 00 00 00 00 00 00 06 d8 05 70 56 a4 00 b0: 10 c4 01 00 c1 0f 00 00 34 48 08 00 81 d4 03 00 c0: 00 00 41 00 05 d4 84 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 11 e0 03 00 00 e0 03 00 00 f0 ff 00 e0: 01 ec 02 02 00 00 00 00 10 00 03 00 03 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Hi Konstantin, this is now a quite old kernel, could you please try to look if a RHEL5.9 kernel shows the problem too? Thanks, Tomas
Hi Tomas, i've explained the problem and showed the place where (most probably) allocations which trigger the problem are done. So the question is - did you fix anything around this place so it really makes sense to try new driver version? Unfortunately the node (if it's still working under 2.6.18-x kernel, cause the bug was filed 3 years(!) ago) was in production and it would be great to have at least a minimal belief that checking (and causing the downtime for server for reboot + possible OOM in the future) is worth to do. Thank you.
(In reply to comment #2) > Hi Tomas, > i've explained the problem and showed the place where (most probably) > allocations which trigger the problem are done. Yes the analysis is great. What are 'Parallels Virtuozzo Containers kernels' and how they are different from our kernels? > > So the question is - did you fix anything around this place so it really > makes sense to try new driver version? Actually, it is hard to say, I can't remember of a specific patch for this issue, but there were so many changes in this driver, that knowing that it touches the latest version really makes sense. > the bug was filed 3 years(!) ago) It is unfortunate^, but it happens sometimes, that nobody notices a bugreport. Please next time contact our support directly, the response times are much better... Tomas
Hi Tomas, (In reply to comment #3) > What are 'Parallels Virtuozzo Containers kernels' and how they are different > from our kernels? Parallels Virtuozzo Containers kernel is a kernel based on Red Hat Enterprise Linux kernel, we take RHEL kernel as a base and add the virtualization part which allows to run isolated Containers on the Node. Usually we do not touch drivers part at all, so in the driver part PVC kernels can be safely considered identical to RHEL's ones. > Actually, it is hard to say, I can't remember of a specific patch for this > issue, but there were so many changes in this driver, that knowing that it > touches the latest version really makes sense. Yes, i agree. Unfortunately the node question was already upgraded to 2.6.32-x based kernel, so no way to reproduce the issue. i believe we can close the issue as obsoleted now. If we ever face this again, we'll file another bug. Thank you.
(In reply to comment #4) > Hi Tomas, > > (In reply to comment #3) > > > What are 'Parallels Virtuozzo Containers kernels' and how they are different > > from our kernels? > > Parallels Virtuozzo Containers kernel is a kernel based on Red Hat > Enterprise Linux kernel, we take RHEL kernel as a base and add the > virtualization part which allows to run isolated Containers on the Node. > > Usually we do not touch drivers part at all, so in the driver part PVC > kernels can be safely considered identical to RHEL's ones. > > > > Actually, it is hard to say, I can't remember of a specific patch for this > > issue, but there were so many changes in this driver, that knowing that it > > touches the latest version really makes sense. > > Yes, i agree. > Unfortunately the node question was already upgraded to 2.6.32-x based > kernel, so no way to reproduce the issue. i believe we can close the issue > as obsoleted now. > If we ever face this again, we'll file another bug. Sure, thank you. I'll close this bug now, further debugging seems to be impossible. Again, thank you for cooperation. > > Thank you.