Bug 560650 - OOM due to probable memory leak in "megaraid_sas" driver
Summary: OOM due to probable memory leak in "megaraid_sas" driver
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.4
Hardware: All
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: Tomas Henzl
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-02-01 14:07 UTC by Konstantin Khorenko
Modified: 2013-03-27 13:46 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-03-27 13:46:09 UTC
Target Upstream Version:
Embargoed:
khorenko: needinfo+


Attachments (Terms of Use)

Description Konstantin Khorenko 2010-02-01 14:07:15 UTC
One of our customers experiences reproducible OOM killer actions on his backup server.

OOM triggers when the node runs Parallels Virtuozzo Containers kernels based on Red Hat 2.6.18-164.2.1.el5 and 2.6.18-164.10.1.el5 kernels (magaraid_sas driver has version 00.00.04.08-RH2).

OOM does not trigger when the node runs PVC kernel based on Red Hat 2.6.18-128.2.1.el5 kernel (magaraid_sas driver has version 00.00.04.01-RH1).

He also tried take a new kernel (based on 2.6.18-164.10.1.el5, which is affected by the issue) and compile older megaraid_sas module from sources for it (v00.00.03.24) - works without problems.


One more note: when running on affected kernels (i'd say on kernels with affected megaraid_sas driver), logs are filled with the following messages:
Dec 14 19:13:40 host kernel: megasas_aen_polling[1]: scanning ...
Dec 14 19:13:42 host kernel:   Vendor: ATA       Model:
ST31000340NS      Rev: MA08
Dec 14 19:13:45 host kernel:   Type:   Direct-Access
             ANSI SCSI revision: 05
Dec 14 19:13:45 host kernel:   Vendor: ATA       Model:
ST31000340NS      Rev: MA08
Dec 14 19:13:48 host kernel:   Type:   Direct-Access
             ANSI SCSI revision: 05
Dec 14 19:13:54 host kernel:   Vendor: ATA       Model:
ST31000340NS      Rev: MA08
Dec 14 19:13:56 host kernel:   Type:   Direct-Access
             ANSI SCSI revision: 05
Dec 14 19:13:58 host kernel:   Vendor: ATA       Model:
ST31000340NS      Rev: MA08
Dec 14 19:14:00 host kernel:   Type:   Direct-Access
             ANSI SCSI revision: 05
-------------------

* Why OOM triggers? Because there is a real memory shortage.
* Who allocated all the memory?
  Alt+SysRQ=M showed that A LOT of memory was used for slab "size-128":
---
size-128             : size 3974189056 objsize        128
---

We've built a debug kernel (CONFIG_DEBUG_SLAB=y, CONFIG_DEBUG_SLAB_LEAK=y) and /proc/slab_allocators file showed that there are abnormally high number of "size-128" objects allocated and not freed in megaraid_sas driver:
---
size-128: 24236892 megasas_complete_cmd_dpc+0x2a6/0x43d [megaraid_sas]
---

This code in question effectively is: megasas_service_aen()->kzalloc().


One more note: the problem occurs on a server with the following controller:
# lspci -s 0e:00.0 -xxx -vvv
0e:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)
        Subsystem: Dell PERC 6/E Adapter RAID Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 169
        Region 0: Memory at fc480000 (64-bit, non-prefetchable) [size=256K]
        Region 2: I/O ports at ec00 [size=256]
        Region 3: Memory at fc440000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at fc300000 [disabled] [size=32K]
        Capabilities: [b0] Express Endpoint IRQ 0
                Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
                Device: Latency L0s unlimited, L1 unlimited
                Device: AtnBtn- AtnInd- PwrInd-
                Device: Errors: Correctable- Non-Fatal- Fatal+ Unsupported-
                Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                Device: MaxPayload 256 bytes, MaxReadReq 2048 bytes
                Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
                Link: Latency L0s <2us, L1 unlimited
                Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
                Link: Speed 2.5Gb/s, Width x8
        Capabilities: [c4] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [d4] MSI-X: Enable- Mask- TabSize=4
                Vector table: BAR=0 offset=0003e000
                PBA: BAR=0 offset=00fff000
        Capabilities: [e0] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [ec] Vital Product Data
        Capabilities: [100] Power Budgeting
00: 00 10 60 00 47 01 18 00 04 00 04 01 10 00 00 00
10: 04 00 48 fc 00 00 00 00 01 ec 00 00 04 00 44 fc
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 0a 1f
30: 00 00 30 fc b0 00 00 00 00 00 00 00 05 01 00 00
40: 00 00 00 00 09 00 00 00 01 00 00 f0 00 00 00 c0
50: 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 05 00 fc ff ff ff ff ff 00 00 00 20 01 00 00 00
80: 01 ff ff ff 00 00 00 00 00 00 00 00 01 00 fc ff
90: ff ff ff ff 00 00 02 00 08 00 00 00 01 80 ff ff
a0: 04 00 00 00 00 00 00 00 00 06 d8 05 70 56 a4 00
b0: 10 c4 01 00 c1 0f 64 00 34 48 08 00 81 d4 03 00
c0: 00 00 81 00 05 d4 84 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 11 e0 03 00 00 e0 03 00 00 f0 ff 00
e0: 01 ec 02 02 00 00 00 00 00 00 0e 00 03 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


The problem DOES NOT occur even with the recent megaraid_sas driver on a node with controller:
# lspci -s 03:00.0 -xxx -vvv
03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)
        Subsystem: Dell PERC 6/i Integrated RAID Controller
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 90
        Region 0: Memory at df180000 (64-bit, non-prefetchable) [size=256K]
        Region 2: I/O ports at fc00 [size=256]
        Region 3: Memory at df1c0000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at df100000 [disabled] [size=32K]
        Capabilities: [b0] Express Endpoint IRQ 0
                Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
                Device: Latency L0s unlimited, L1 unlimited
                Device: AtnBtn- AtnInd- PwrInd-
                Device: Errors: Correctable- Non-Fatal- Fatal+ Unsupported-
                Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                Device: MaxPayload 256 bytes, MaxReadReq 2048 bytes
                Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
                Link: Latency L0s <2us, L1 unlimited
                Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
                Link: Speed 2.5Gb/s, Width x4
        Capabilities: [c4] Message Signalled Interrupts: 64bit+ Queue=0/2 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [d4] MSI-X: Enable- Mask- TabSize=4
                Vector table: BAR=0 offset=0003e000
                PBA: BAR=0 offset=00fff000
        Capabilities: [e0] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [ec] Vital Product Data
        Capabilities: [100] Power Budgeting
00: 00 10 60 00 07 00 10 00 04 00 04 01 10 00 00 00
10: 04 00 18 df 00 00 00 00 01 fc 00 00 04 00 1c df
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 0c 1f
30: 00 00 10 df b0 00 00 00 00 00 00 00 0f 01 00 00
40: 00 00 00 00 09 00 00 00 01 00 00 f0 00 00 00 c0
50: 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 05 00 fc ff ff ff ff ff 00 00 00 20 01 00 00 00
80: 01 ff ff ff 00 00 00 00 00 00 00 00 01 00 fc ff
90: ff ff ff ff 00 00 02 00 08 00 00 00 01 80 ff ff
a0: 04 00 00 00 00 00 00 00 00 06 d8 05 70 56 a4 00
b0: 10 c4 01 00 c1 0f 00 00 34 48 08 00 81 d4 03 00
c0: 00 00 41 00 05 d4 84 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 11 e0 03 00 00 e0 03 00 00 f0 ff 00
e0: 01 ec 02 02 00 00 00 00 10 00 03 00 03 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Comment 1 Tomas Henzl 2013-03-20 13:21:03 UTC
Hi Konstantin,
this is now a quite old kernel, could you please try to look if a RHEL5.9 kernel shows the problem too?
Thanks,
Tomas

Comment 2 Konstantin Khorenko 2013-03-22 10:56:50 UTC
Hi Tomas,
i've explained the problem and showed the place where (most probably) allocations which trigger the problem are done.

So the question is - did you fix anything around this place so it really makes sense to try new driver version?

Unfortunately the node (if it's still working under 2.6.18-x kernel, cause the bug was filed 3 years(!) ago) was in production and it would be great to have at least a minimal belief that checking (and causing the downtime for server for reboot + possible OOM in the future) is worth to do.

Thank you.

Comment 3 Tomas Henzl 2013-03-22 15:00:29 UTC
(In reply to comment #2)
> Hi Tomas,
> i've explained the problem and showed the place where (most probably)
> allocations which trigger the problem are done.

Yes the analysis is great. 
What are 'Parallels Virtuozzo Containers kernels' and how they are different from our kernels? 
> 
> So the question is - did you fix anything around this place so it really
> makes sense to try new driver version?

Actually, it is hard to say, I can't remember of a specific patch for this issue, but there were so many changes in this driver, that knowing that it touches the latest version really makes sense.

> the bug was filed 3 years(!) ago)
It is unfortunate^, but it happens sometimes, that nobody notices a bugreport. Please next time contact our support directly, the response times are much better...

Tomas

Comment 4 Konstantin Khorenko 2013-03-25 09:21:20 UTC
Hi Tomas,

(In reply to comment #3)

> What are 'Parallels Virtuozzo Containers kernels' and how they are different
> from our kernels? 

Parallels Virtuozzo Containers kernel is a kernel based on Red Hat Enterprise Linux kernel, we take RHEL kernel as a base and add the virtualization part which allows to run isolated Containers on the Node.

Usually we do not touch drivers part at all, so in the driver part PVC kernels can be safely considered identical to RHEL's ones.


> Actually, it is hard to say, I can't remember of a specific patch for this
> issue, but there were so many changes in this driver, that knowing that it
> touches the latest version really makes sense.

Yes, i agree.
Unfortunately the node question was already upgraded to 2.6.32-x based kernel, so no way to reproduce the issue. i believe we can close the issue as obsoleted now.
If we ever face this again, we'll file another bug.

Thank you.

Comment 5 Tomas Henzl 2013-03-27 13:46:09 UTC
(In reply to comment #4)
> Hi Tomas,
> 
> (In reply to comment #3)
> 
> > What are 'Parallels Virtuozzo Containers kernels' and how they are different
> > from our kernels? 
> 
> Parallels Virtuozzo Containers kernel is a kernel based on Red Hat
> Enterprise Linux kernel, we take RHEL kernel as a base and add the
> virtualization part which allows to run isolated Containers on the Node.
> 
> Usually we do not touch drivers part at all, so in the driver part PVC
> kernels can be safely considered identical to RHEL's ones.
> 
> 
> > Actually, it is hard to say, I can't remember of a specific patch for this
> > issue, but there were so many changes in this driver, that knowing that it
> > touches the latest version really makes sense.
> 
> Yes, i agree.
> Unfortunately the node question was already upgraded to 2.6.32-x based
> kernel, so no way to reproduce the issue. i believe we can close the issue
> as obsoleted now.
> If we ever face this again, we'll file another bug.
Sure, thank you. 
I'll close this bug now, further debugging seems to be impossible.
Again, thank you for cooperation.
> 
> Thank you.


Note You need to log in before you can comment on or make changes to this bug.