Hardware testing for the Mellanox MT25204 has revealed that an internal error
occurs under certain high-load conditions. When the ib_mthca driver reports a
catastrophic error on this hardware, it is usually related to an insufficient
completion queue depth relative to the number of outstanding work requests
generated by the user application.
Although the driver will reset the hardware and recover from such an event, all
existing connections at the time of the error will be lost. This generally
results in a segmentation fault in the user application. Further, if opensm is
running at the time the error occurs, then you need to manually restart it in
order to resume proper operation.