Well the coredump certainly tells me that the memory has gotten corrupted, but it would be helpful to have /var/log/messages from a run where this happens to see what is going on immediately before the crash.

Also, have you tried this with recent versions of device-mapper-multipath? While this doesn't look exactly like any issue I remember, there have been memory issues fixed since RHEL-6.3.

Well, the corrupt memory looks like this:

$49 = "0000000f188b10030\000\000\000\000\000\000\000$\n\000\000\000\000\000\000\320\020\016X\325\177\000\000\000/\016X\325\177\000\000\020/\016X\325\177\000\000/devices/pci0000:00/0000:00:09.0/0000:10:00.0/0000:11:03.0/0000:13:00.0/host15/rport-15:0-17/target15:0:12/15:0:12:62/block/sdct", '\000' <repeats 315 times>

Which should help locating the problem.

(In reply to Ben Marzinski from comment #2)
> Well the coredump certainly tells me that the memory has gotten corrupted,
> but it would be helpful to have /var/log/messages from a run where this
> happens to see what is going on immediately before the crash.

I will get someone to reproduce this and gather the var/log/messages. Would you like anything else? Perhaps enhanced multipathd logging? What version of RHEL and multipath would you like us to run? This happens fairly consistently with all versions of RHEL we tested.

well, looking at the corrupt memory itself doesn't appear horribly helpful.  Actually, the only bit of data that's left in the corrupted malloc chunk (besides the data that overwrote the linked list pointer with 0x0) is "0000000f188b1003".  That's pretty definitely part of the WWID from one of your paths, judging from the complete ones I found in the memory dump.

If this was a use-after-free corruption, then something would have to be writing
0x0 to the first 8 bytes of that wwid after it was freed.  Looking through the code I can't find any obvious place where this could be happening. However, I only looked at variables that definitely store the wwid and get freed.  If you aren't using user_friendly_names, then the multipath's alias is also set to the wwid.  So, is user_friendly_names set in your /etc/multipath.conf?

Also, I didn't look at the the code for the multipaths section of /etc/multipath.conf.  The multipath entries their are identified by wwid. If you have a multipaths section in your /etc/multipath.conf, and during your test you reload the multipathd configuration using

# service multipathd reload

or

# multipathd reconfigure

then these would also be freed, and it would be worth checking there.

So, I should probably take a look at your /etc/multipath.conf file. Could you please attach that to the bugzilla?

Another possibility is that the previous malloc chunk overwrote it's allotted memory, and corrupted this entry.  But the previous entry is a path structure, and it looks fine, so this isn't likely. Also the beginning of the malloc
structure is fine.  The only part got overwritten is the part that is used
by the actual data when the chunk is in-use.

Right now my best guess is that this is a use-after-free issue, but whatever old memory is being used has been freed for a while, and the space has been already reused to store the wwid.

However, there are some weirdnesses I've noticed.

When I start up gdb, I get the following message:

Missing separate debuginfo for /lib64/multipath/libpriosfa.so

We don't distribute a libpriosfa.so DSO. I found the hardware entry for your paths in the memory dump, and they are using this prio function. Is this something you wrote? If it is, it's possible that the corruption is in there, and I would have no way to track that down without the source code.

Another weirdness is that it seems pretty certain that the device-mapper code is requesting a 40 byte piece of memory.  In the backtrace, I see

#5  0x000000348847a911 in __libc_malloc (bytes=40) at malloc.c:3664
#6  0x0000003489421656 in dm_zalloc_aux (s=40, file=<value optimized out>,
    line=<value optimized out>) at mm/dbg_malloc.c:274

However, the chuck that the malloc code is checking is only 48 bytes, and that includes the 16 bytes of malloc overhead, so it could only be returning a 32 byte chunk to the caller. I'm not sure what to make of this. I rarely go digging through the malloc internals to try to track down a bug so I could just be misunderstanding something, but examining the structures in GDB certainly makes it appear that malloc overhead is 16 bytes on an x86_64 system (it's 8 on an i686 system).  At any rate, this isn't what caused multipath to crash.  The overwritten pointer is 16 bytes in, and the earlier part of the malloc structure is fine (the malloc chunk structure uses 48 bytes for freed chunks). Also, I have a lot of problems believing that malloc could be returning the wrong size memory without causing problems much sooner.  I just though I'd mention it.

The version of the tools I'd like you to test is available here:

http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/RHEL6/1012672/

This has a fix that hasn't made it into a release yet, for a problem that can cause memory corruption.  If you never remove any devices before multipathd crashes in your tests, then I don't think that specific fix will help you, but it is the most uptodate code available.

Created attachment 848235 [details]
multipath.conf

(In reply to Ben Marzinski from comment #6)
> well, looking at the corrupt memory itself doesn't appear horribly helpful. 
> Actually, the only bit of data that's left in the corrupted malloc chunk
> (besides the data that overwrote the linked list pointer with 0x0) is
> "0000000f188b1003".  That's pretty definitely part of the WWID from one of
> your paths, judging from the complete ones I found in the memory dump.
> 
> If this was a use-after-free corruption, then something would have to be
> writing
> 0x0 to the first 8 bytes of that wwid after it was freed.  Looking through
> the code I can't find any obvious place where this could be happening.
> However, I only looked at variables that definitely store the wwid and get
> freed.  If you aren't using user_friendly_names, then the multipath's alias
> is also set to the wwid.  So, is user_friendly_names set in your
> /etc/multipath.conf?
> 
> Also, I didn't look at the the code for the multipaths section of
> /etc/multipath.conf.  The multipath entries their are identified by wwid. If
> you have a multipaths section in your /etc/multipath.conf, and during your
> test you reload the multipathd configuration using
> 
> # service multipathd reload
> 
> or
> 
> # multipathd reconfigure
> 
> then these would also be freed, and it would be worth checking there.
> 
> So, I should probably take a look at your /etc/multipath.conf file. Could
> you please attach that to the bugzilla?
> 
> Another possibility is that the previous malloc chunk overwrote it's
> allotted memory, and corrupted this entry.  But the previous entry is a path
> structure, and it looks fine, so this isn't likely. Also the beginning of
> the malloc
> structure is fine.  The only part got overwritten is the part that is used
> by the actual data when the chunk is in-use.
> 
> Right now my best guess is that this is a use-after-free issue, but whatever
> old memory is being used has been freed for a while, and the space has been
> already reused to store the wwid.
> 
> However, there are some weirdnesses I've noticed.
> 
> When I start up gdb, I get the following message:
> 
> Missing separate debuginfo for /lib64/multipath/libpriosfa.so
> 
> We don't distribute a libpriosfa.so DSO. I found the hardware entry for your
> paths in the memory dump, and they are using this prio function. Is this
> something you wrote? If it is, it's possible that the corruption is in
> there, and I would have no way to track that down without the source code.
> 
> Another weirdness is that it seems pretty certain that the device-mapper
> code is requesting a 40 byte piece of memory.  In the backtrace, I see
> 
> #5  0x000000348847a911 in __libc_malloc (bytes=40) at malloc.c:3664
> #6  0x0000003489421656 in dm_zalloc_aux (s=40, file=<value optimized out>,
>     line=<value optimized out>) at mm/dbg_malloc.c:274
> 
> However, the chuck that the malloc code is checking is only 48 bytes, and
> that includes the 16 bytes of malloc overhead, so it could only be returning
> a 32 byte chunk to the caller. I'm not sure what to make of this. I rarely
> go digging through the malloc internals to try to track down a bug so I
> could just be misunderstanding something, but examining the structures in
> GDB certainly makes it appear that malloc overhead is 16 bytes on an x86_64
> system (it's 8 on an i686 system).  At any rate, this isn't what caused
> multipath to crash.  The overwritten pointer is 16 bytes in, and the earlier
> part of the malloc structure is fine (the malloc chunk structure uses 48
> bytes for freed chunks). Also, I have a lot of problems believing that
> malloc could be returning the wrong size memory without causing problems
> much sooner.  I just though I'd mention it.
> 
> The version of the tools I'd like you to test is available here:
> 
> http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/RHEL6/1012672/
> 
> This has a fix that hasn't made it into a release yet, for a problem that
> can cause memory corruption.  If you never remove any devices before
> multipathd crashes in your tests, then I don't think that specific fix will
> help you, but it is the most uptodate code available.

User friendly names are not set. I have attached the multipath.conf to this bug report. 

I am sure at some point we must have issued 'service multipathd restart', but that must have happened before the test started.

libpriosfa.so is something we wrote. However, to rule out memory corruption because of it, we reproduced the same problem with prioritizer set to "alua". Also, we can provide the source code for libpriosafa if you like.

We will try this test out with the latest version as you specified in your comment.

Thanks for looking at this so quickly!

(In reply to Ben Marzinski from comment #6)
> well, looking at the corrupt memory itself doesn't appear horribly helpful. 
> Actually, the only bit of data that's left in the corrupted malloc chunk
> (besides the data that overwrote the linked list pointer with 0x0) is
> "0000000f188b1003".  That's pretty definitely part of the WWID from one of
> your paths, judging from the complete ones I found in the memory dump.
> 
> If this was a use-after-free corruption, then something would have to be
> writing
> 0x0 to the first 8 bytes of that wwid after it was freed.  Looking through
> the code I can't find any obvious place where this could be happening.
> However, I only looked at variables that definitely store the wwid and get
> freed.  If you aren't using user_friendly_names, then the multipath's alias
> is also set to the wwid.  So, is user_friendly_names set in your
> /etc/multipath.conf?
> 
> Also, I didn't look at the the code for the multipaths section of
> /etc/multipath.conf.  The multipath entries their are identified by wwid. If
> you have a multipaths section in your /etc/multipath.conf, and during your
> test you reload the multipathd configuration using
> 
> # service multipathd reload
> 
> or
> 
> # multipathd reconfigure
> 
> then these would also be freed, and it would be worth checking there.
> 
> So, I should probably take a look at your /etc/multipath.conf file. Could
> you please attach that to the bugzilla?
> 
> Another possibility is that the previous malloc chunk overwrote it's
> allotted memory, and corrupted this entry.  But the previous entry is a path
> structure, and it looks fine, so this isn't likely. Also the beginning of
> the malloc
> structure is fine.  The only part got overwritten is the part that is used
> by the actual data when the chunk is in-use.
> 
> Right now my best guess is that this is a use-after-free issue, but whatever
> old memory is being used has been freed for a while, and the space has been
> already reused to store the wwid.
> 
> However, there are some weirdnesses I've noticed.
> 
> When I start up gdb, I get the following message:
> 
> Missing separate debuginfo for /lib64/multipath/libpriosfa.so
> 
> We don't distribute a libpriosfa.so DSO. I found the hardware entry for your
> paths in the memory dump, and they are using this prio function. Is this
> something you wrote? If it is, it's possible that the corruption is in
> there, and I would have no way to track that down without the source code.
> 
> Another weirdness is that it seems pretty certain that the device-mapper
> code is requesting a 40 byte piece of memory.  In the backtrace, I see
> 
> #5  0x000000348847a911 in __libc_malloc (bytes=40) at malloc.c:3664
> #6  0x0000003489421656 in dm_zalloc_aux (s=40, file=<value optimized out>,
>     line=<value optimized out>) at mm/dbg_malloc.c:274
> 
> However, the chuck that the malloc code is checking is only 48 bytes, and
> that includes the 16 bytes of malloc overhead, so it could only be returning
> a 32 byte chunk to the caller. I'm not sure what to make of this. I rarely
> go digging through the malloc internals to try to track down a bug so I
> could just be misunderstanding something, but examining the structures in
> GDB certainly makes it appear that malloc overhead is 16 bytes on an x86_64
> system (it's 8 on an i686 system).  At any rate, this isn't what caused
> multipath to crash.  The overwritten pointer is 16 bytes in, and the earlier
> part of the malloc structure is fine (the malloc chunk structure uses 48
> bytes for freed chunks). Also, I have a lot of problems believing that
> malloc could be returning the wrong size memory without causing problems
> much sooner.  I just though I'd mention it.
> 
> The version of the tools I'd like you to test is available here:
> 
> http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/RHEL6/1012672/
> 
> This has a fix that hasn't made it into a release yet, for a problem that
> can cause memory corruption.  If you never remove any devices before
> multipathd crashes in your tests, then I don't think that specific fix will
> help you, but it is the most uptodate code available.

Ben, your RHEL6 rpms seem to be helping the situation. Do you have RHEL5 rpms as well that we could test?

Have you seen this issue on RHEL5?  There are some significant differences in the code. Without figuring out which change appears to have fixed your issue, I can't really know if the effected code even exists in RHEL5.  What is the most recent RHEL6 package you tried and had fail?

1. Was the test files your provided .. added to the released versions of 
        device-mapper-multipath-libs-0.4.9-72...
        device-mapper-multipath-0.4.9-72...

2.  If your patched file were on the actual 0.4.9-72 files,  then what would be an anticipated release timeframe ?

3.  what is future for this problem on RH5 ?    ??do you need specific bits from RH5 failure ?

thanks,

I accidentally made comment #10 private.  Sorry.

The patches from

http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/RHEL6/1012672/

Will be going into the RHEL6.6 release. There will most likely be a RHEL-6.5 zstream release with the fix.  The decision on that is currently waiting on the customer to verify that the fix does resolve their issue. However, if these patches have fixed your memory corruption issue then it getting a zstream release shouldn't be a problem.

I don't have a good idea on the timeframe for a zstream release.  Since the patch is already written, my part of the whole process would take about an hour or so.  The bigger unknown is QA's schedule.

Like I mentioned in Comment #10, I'd need to know which fix solved your issue.  If it was solved by the fix for 1012672, then it's not in RHEL5. In RHEL5, none of the sysfs handling code that 1012672 fixed is in multipath.  Multipath relies on libsysfs, and there is no analog to the RHEL6 code that has the issue.

(In reply to Ben Marzinski from comment #10)
> Have you seen this issue on RHEL5?  There are some significant differences
> in the code. Without figuring out which change appears to have fixed your
> issue, I can't really know if the effected code even exists in RHEL5.  What
> is the most recent RHEL6 package you tried and had fail?

The following RH5  and version of multipath bits as noted below haved shown symptoms of  multipathd service locking up and you have to reboot server.  A service stop/start or restart are not effective in clearing the condition.
 
================================================================================10.36.30.46"  ==RH5.9
device-mapper-multipath-0.4.7-54.el5_9.2
================================================================================10.36.30.50"==RH5.10
device-mapper-multipath-0.4.7-59.el5
 ================================================================================10.36.30.51"==RH5.9
device-mapper-multipath-0.4.7-54.el5_9.1
 ================================================================================10.36.30.52"==RH5.10
device-mapper-multipath-0.4.7-59.el5
================================================================================10.36.30.55"==RH5.9
device-mapper-multipath-0.4.7-54.el5_9.1
 ================================================================================10.36.30.98"==RH5.10
device-mapper-multipath-0.4.7-59.el5
 ================================================================================

My RH6 stations (6.3,6.4,6.5) with your test bits continue to NOT exhibit this symptom, but my remaining RH6 stations that had the distro-released multipath versions (6.3,6.4) continue to consistently lock up.

(In reply to Ben Marzinski from comment #12)
> I accidentally made comment #10 private.  Sorry.
> 
> The patches from
> 
> http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/RHEL6/1012672/
> 
> Will be going into the RHEL6.6 release. There will most likely be a RHEL-6.5
> zstream release with the fix.  The decision on that is currently waiting on
> the customer to verify that the fix does resolve their issue. However, if
> these patches have fixed your memory corruption issue then it getting a
> zstream release shouldn't be a problem.
> 
> I don't have a good idea on the timeframe for a zstream release.  Since the
> patch is already written, my part of the whole process would take about an
> hour or so.  The bigger unknown is QA's schedule.
> 
> Like I mentioned in Comment #10, I'd need to know which fix solved your
> issue.  If it was solved by the fix for 1012672, then it's not in RHEL5. In
> RHEL5, none of the sysfs handling code that 1012672 fixed is in multipath. 
> Multipath relies on libsysfs, and there is no analog to the RHEL6 code that
> has the issue.

Hi Ben,

Attached is a core file from a rhel 5.9 multipathd crash.


device-mapper-multipath-0.4.7-54.el5_9.2
device-mapper-multipath-debuginfo-0.4.7-59.el5
device-mapper-1.02.67-2.el5
device-mapper-multipath-0.4.7-54.el5_9.2
device-mapper-1.02.67-2.el5
device-mapper-debuginfo-1.02.67-2.el5
device-mapper-event-1.02.67-2.el5
device-mapper-multipath-debuginfo-0.4.7-59.el5

Do we have a fix for this?

Thanks
Karan

Created attachment 857949 [details]
RHEL 5.9 multipathd segfault

I don't recall ever seeing this issue before, but I'll take a look though the bugs fixed since rhel-5.9.  If I don't find anything, you should probably open a separate bugzilla, since this does not look like the issue fixed by the latest rhel-6 code.

Filed a new defect for RHEL 5.9 bug=1064409

*** Bug 1012672 has been marked as a duplicate of this bug. ***

Hi Wang,

Exactly what information do you need? We ran many days with the fix provided by Ben and did not encounter any problems. We ran on a big SAN with hundreds of FC initiators and storage. The initiators were mainly RHEL 6x machines.

Please let me know exactly what information is required.

Thanks
Karan

Added the fix from the test package

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1555.html