Bug 1579492

Summary: Race condition leads to report that index in memory is corrupt
Product: Red Hat Enterprise Linux 7 Reporter: Thomas Jaskiewicz <tjaskiew>
Component: kmod-kvdoAssignee: Thomas Jaskiewicz <tjaskiew>
Status: CLOSED ERRATA QA Contact: Jakub Krysl <jkrysl>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.6CC: awalsh, jkrysl, limershe
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 6.1.1.94 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-30 09:39:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Thomas Jaskiewicz 2018-05-17 18:43:18 UTC 
Description of problem:

This is a problem we have been seeing in our nightly testing.  We do not have a way of reproducing the failure, but once or twice a month we see this kind of report:

Apr  2 09:33:00 parkst-31 kernel: [25652.104178] uds: zub:indexW: assertion "Delta list number (173) is out of range (171)" ((listNumber < deltaIndex->numLists)) failed at /var/lib/dkms/zubenelgenubi/1/build/uds/deltaIndex.c:1062
...
Apr  2 09:33:00 parkst-31 kernel: [25652.104326] uds: zub:indexW: index_0: searchLockedIndex() failed: UDS Error: Index data in memory is corrupt (1036)

Our RHEL8 work has required us to examine our multithreading of our code.  Historically we have coded to work on x86 hardware, but for RHEL8 we need to pay attention to proper use of atomic values and memory barriers.  As part of this work, we have found data race conditions in the UDS code that can lead to the failure asserted above in kern.log.


Version-Release number of selected component (if applicable):

All released versions of kmod-kvdo have these problems.


How reproducible:

Very difficult.  We see failures once or twice a month in our nightly testing.  The actual races were found by inspecting the source code.


Steps to Reproduce:
1. ??
2. ??

Actual results:
Expected results:

Obviouely no report of corruption.  The best suggestion is to run sanity tests and any basic operational testing that QA has developed.


Additional info:

We want to merge the RHEL8 work related to these races back to RHEL7.6.  It is a large change, but we have been testing it nightly in our RHEL8 code stream.
Comment 3 Jakub Krysl 2018-07-09 13:41:20 UTC 
Regression testing passed on:
RHEL-7.6-20180626.0
kernel-3.10.0-915.el7
kmod-vdo-6.1.1.99-1.el7
vdo-6.1.1.99-2.el7

As there is no known way to reproduce this, setting to verified with SanityOnly.
Comment 5 errata-xmlrpc 2018-10-30 09:39:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3094