Bug 1934715

Summary:	DB corruption "_entryrdn_insert_key - Same DN (dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,<SUFFIX>) is already in the entryrdn file"
Product:	Red Hat Directory Server	Reporter:	Têko Mihinto <tmihinto>
Component:	389-ds-base	Assignee:	mreynolds
Status:	CLOSED ERRATA	QA Contact:	RHDS QE <ds-qe-bugs>
Severity:	high	Docs Contact:	Marc Muehlfeld <mmuehlfe>
Priority:	high
Version:	11.3	CC:	afarley, asakure, asharov, bsmejkal, gerald.prock, gkimetto, ldap-maint, ldelouw, mreynolds, mrhodes, msauton, pcech, sgouvern, tbordaz
Target Milestone:	DS11.3	Keywords:	TestCaseProvided, Triaged
Target Release:	dirsrv-11.6
Hardware:	x86_64
OS:	Linux
Whiteboard:	sync-to-jira
Fixed In Version:	redhat-ds-11-8050020210917175916.d3df4063	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2022686 2024695 (view as bug list)		Environment:
Last Closed:	2022-11-14 08:53:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2022686, 2024695

Description Têko Mihinto 2021-03-03 17:33:23 UTC

Description of problem:

There seems to be a DB corruption issue with the entry nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,<SUFFIX>.
Typically after a reindexing of a replicated suffix.

Errors log:
[25/Feb/2021:10:34:40.147633966 +0100] - INFO - bdb_db2index - userroot: Finished indexing.
[25/Feb/2021:10:34:42.199441489 +0100] - ERR - _entryrdn_insert_key - Same DN (dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com) is already in the entryrdn file with different ID 133529.  Expected ID is 133541.
[25/Feb/2021:10:34:42.202046664 +0100] - ERR - index_addordel_entry - database index operation failed BAD 1023, err=9999 Unknown error 9999
[25/Feb/2021:10:34:42.220902975 +0100] - ERR - NSMMReplicationPlugin - _replica_configure_ruv - Failed to create replica ruv tombstone entry (dc=example,dc=com); LDAP error - 1
[25/Feb/2021:10:35:12.240676559 +0100] - ERR - _entryrdn_insert_key - Same DN (dn: nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff,dc=example,dc=com) is already in the entryrdn file with different ID 133529.  Expected ID is 133541.

Version-Release number of selected component (if applicable):

$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)
$
$ grep ^389-ds installed-rpms
389-ds-base-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64  Tue Jan  5 10:59:38 2021
389-ds-base-debuginfo-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Mon Feb  8 12:25:14 2021
389-ds-base-debugsource-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Mon Feb  8 12:25:13 2021
389-ds-base-legacy-tools-debuginfo-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Mon Feb  8 12:25:15 2021
389-ds-base-libs-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Tue Jan  5 10:59:38 2021
389-ds-base-libs-debuginfo-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Mon Feb  8 12:25:15 2021
389-ds-base-snmp-debuginfo-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Mon Feb  8 12:25:16 2021
$

How reproducible:
At customer site.

Steps to Reproduce:
Triggered by a reindexing of a replicated suffix.

Actual results:
corrupt DB.

$ grep -c "is already in the entryrdn file with different ID" errors* | grep -v :0$
errors:12388
errors.20210208-125224:3
errors.20210216-113244:6401
$

Expected results:
Sane DB.

Additional info:
Similar issue described here:
    https://www.spinics.net/linux/fedora/389-users/msg21664.html

Comment 10 mreynolds 2021-09-07 17:43:18 UTC

I've been able to reproduce this using a CI test that repeatedly makes a few updates, and reindexes the database.  It took a few minutes, but I was able to reproduce it.  Continuing investigation...

Comment 11 mreynolds 2021-09-08 14:42:01 UTC

Root cause identified and fixed in:

https://github.com/389ds/389-ds-base/issues/4910

Comment 12 mreynolds 2021-09-09 11:52:04 UTC

Fixed upstream.

Comment 13 Marc Sauton 2021-09-09 12:17:56 UTC

thanks for the analysis and fix, will try to prepare a hotfix for RHDS-11.3 customers, will need RHDS-11.4 errata

Comment 15 Petr Čech 2022-05-25 12:16:14 UTC

As the RHEL 8.5 reached EOL, it makes sense to move this BZ to dirsrv-11.5, which means RHEL 8.6.

Comment 18 sgouvern 2022-07-27 13:08:15 UTC

waiting for a new build with https://github.com/389ds/389-ds-base/pull/5086 backported in it 
-> moving back from MODIFIED to ASSIGNED

Comment 25 errata-xmlrpc 2022-11-14 08:53:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-ds:11 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7929