RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1402012 - Importing big ldif file with duplicate DNs throwing "unable to flush" error
Summary: Importing big ldif file with duplicate DNs throwing "unable to flush" error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: 389-ds-base
Version: 6.9
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: mreynolds
QA Contact: Viktor Ashirov
Marc Muehlfeld
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-06 15:05 UTC by Sankar Ramalingam
Modified: 2020-09-13 21:54 UTC (History)
6 users (show)

Fixed In Version: 389-ds-base-1.2.11.15-86.el6
Doc Type: Bug Fix
Doc Text:
Directory Server no longer displays multiple error messages when importing fails Previously, if importing data failed, multiple "Unable to flush" error message were be displayed, because the connection to the database was not closed. This update applies a patch and as a result, Directory Server no longer displays multiple errors in the mentioned situation.
Clone Of:
Environment:
Last Closed: 2017-03-21 10:24:12 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 2130 0 None closed Importing big ldif file with duplicate DNs throwing "unable to flush" error 2021-02-15 03:53:20 UTC
Red Hat Product Errata RHBA-2017:0667 0 normal SHIPPED_LIVE 389-ds-base bug fix update 2017-03-21 12:35:05 UTC

Comment 6 Sankar Ramalingam 2016-12-15 16:26:10 UTC 
Hi Noriko, I tried with a smaller LDIF file and outcome is almost the same. I will try again with fewer no of entries and report them here.
cat /var/lib/dirsrv/slapd-Inst2/ldif/100k.ldif |egrep 'dn: uid|dn: cn' |wc -l
10828

[root@vm-idm-029 ldif]# /usr/lib64/dirsrv/slapd-Inst2/ldif2db.pl -D "cn=Directory Manager" -w Secret123  -s "dc=example,dc=com" -n userRoot -i /var/lib/dirsrv/slapd-Inst2/ldif/100k.ldif 
Successfully added task entry "cn=import_2016_12_15_21_53_53, cn=import, cn=tasks, cn=config"
[root@vm-idm-029 ldif]# tail -f /var/log/dirsrv/slapd-Inst2/errors[15/Dec/2016:21:53:54.307568694 +051800] import userRoot: Duplicated DN detected: "uid=tmorris,ou=People,dc=example,dc=com": Entry ID: (25)
[15/Dec/2016:21:53:54.354383450 +051800] import userRoot: Thread monitoring returned: -23

[15/Dec/2016:21:53:54.375463443 +051800] import userRoot: Aborting all Import threads...
[15/Dec/2016:21:53:59.889652750 +051800] import userRoot: Import threads aborted.
[15/Dec/2016:21:53:59.893392342 +051800] import userRoot: Closing files...
[15/Dec/2016:21:53:59.932450268 +051800] libdb: BDB3028 userRoot/entryrdn.db: unable to flush: No such file or directory
[15/Dec/2016:21:53:59.940701237 +051800] libdb: BDB3028 userRoot/id2entry.db: unable to flush: No such file or directory
[15/Dec/2016:21:53:59.949394528 +051800] libdb: BDB3028 userRoot/parentid.db: unable to flush: No such file or directory
[15/Dec/2016:21:53:59.952590866 +051800] import userRoot: Import failed.
Comment 7 mreynolds 2016-12-19 17:08:26 UTC 
I was able to reproduce this with a 3 entry ldif file:

dn: dc=example,dc=com
objectclass: top
objectclass: domain
dc: example

dn: ou=myDups00001,dc=example,dc=com
objectclass: top
objectclass: organizationalUnit
ou: myDups00001

dn: ou=myDups00001,dc=example,dc=com
objectclass: top
objectclass: organizationalUnit
ou: myDups00001


On master branch I see:

[19/Dec/2016:12:05:06.356574675 -0500] - INFO - import_producer - import userroot: Processing file "/tmp/out.ldif"
[19/Dec/2016:12:05:06.562994763 -0500] - ERR - _entryrdn_insert_key - Same DN (dn: ou=myDups00001,dc=example,dc=com) is already in the entryrdn file with different ID 2.  Expected ID is 3.
[19/Dec/2016:12:05:06.565045265 -0500] - ERR - foreman_do_entryrdn - import userroot: Duplicated DN detected: "ou=myDups00001,dc=example,dc=com": Entry ID: (3)
[19/Dec/2016:12:05:06.658642236 -0500] - ERR - import_run_pass - import userroot: Thread monitoring returned: -23

[19/Dec/2016:12:05:06.660725660 -0500] - ERR - import_main_offline - import userroot: Aborting all Import threads...
[19/Dec/2016:12:05:12.168287348 -0500] - ERR - import_main_offline - import userroot: Import threads aborted.
[19/Dec/2016:12:05:12.170243072 -0500] - INFO - import_main_offline - import userroot: Closing files...
[19/Dec/2016:12:05:12.184798674 -0500] - ERR - libdb - BDB3028 userroot/entryrdn.db: unable to flush: No such file or directory
[19/Dec/2016:12:05:12.186952872 -0500] - ERR - libdb - BDB3028 userroot/id2entry.db: unable to flush: No such file or directory
[19/Dec/2016:12:05:12.188989348 -0500] - ERR - libdb - BDB3028 userroot/parentid.db: unable to flush: No such file or directory
/var/lib/dirsrv/slapd-localhost/db/userroot: No such file or directory
[19/Dec/2016:12:05:12.203029854 -0500] - INFO - dblayer_pre_close - All database threads now stopped
[19/Dec/2016:12:05:12.205085596 -0500] - ERR - import_main_offline - import userroot: Import failed.
Comment 8 mreynolds 2016-12-19 17:25:55 UTC 
Upstream ticket:
https://fedorahosted.org/389/ticket/49071
Comment 9 mreynolds 2016-12-19 17:42:26 UTC 
There was a bug in DS where we were deleting the backend files before we closed the database down - this is what caused those unexpected errors.

Fixed upstream.
Comment 11 Sankar Ramalingam 2017-01-11 11:01:23 UTC 
Unable to flush error is gone now with the latest build of 389-ds-base-1.2.11.15-86. However, the suffix/database cannot be reached for further ldap operations. I did also notice that the import task is still running and cannot be deleted. I believe the import task should exit as soon as it says "Aborting all import threads".

[root@cypher 389ds-replica]# /usr/lib64/dirsrv/slapd-Inst1/ldif2db.pl -D "cn=Directory Manager" -w Secret123 -n importest1121 -s "dc=importest,dc=com" -i /var/lib/dirsrv/slapd-Inst1/ldif/MyNew02_01.ldif
adding new entry "cn=import_2017_1_11_4_2_35, cn=import, cn=tasks, cn=config"

[root@cypher export]# tail -f /var/log/dirsrv/slapd-Inst1/errors
[11/Jan/2017:04:02:35 -0500] - import importest1121: Beginning import job...
[11/Jan/2017:04:02:35 -0500] - import importest1121: Index buffering enabled with bucket size 19
[11/Jan/2017:04:02:35 -0500] - import importest1121: Processing file "/var/lib/dirsrv/slapd-Inst1/ldif/MyNew02_01.ldif"
[11/Jan/2017:04:02:55 -0500] - import importest1121: Processed 12319 entries -- average rate 616.0/sec, recent rate 615.9/sec, hit ratio 0%
[11/Jan/2017:04:03:15 -0500] - import importest1121: Processed 22870 entries -- average rate 571.8/sec, recent rate 571.7/sec, hit ratio 100%
[11/Jan/2017:04:03:36 -0500] - import importest1121: Processed 33687 entries -- average rate 561.4/sec, recent rate 534.2/sec, hit ratio 100%
[11/Jan/2017:04:03:56 -0500] - import importest1121: Processed 44062 entries -- average rate 550.7/sec, recent rate 529.7/sec, hit ratio 100%
[11/Jan/2017:04:04:16 -0500] - import importest1121: Processed 54003 entries -- average rate 534.7/sec, recent rate 495.5/sec, hit ratio 100%
[11/Jan/2017:04:04:42 -0500] - import importest1121: Processed 67035 entries -- average rate 527.8/sec, recent rate 488.9/sec, hit ratio 100%
[11/Jan/2017:04:05:02 -0500] - import importest1121: Processed 76844 entries -- average rate 522.7/sec, recent rate 496.5/sec, hit ratio 99%
[11/Jan/2017:04:05:22 -0500] - import importest1121: Processed 86641 entries -- average rate 518.8/sec, recent rate 490.1/sec, hit ratio 99%
[11/Jan/2017:04:05:38 -0500] - ldbm: 'importest1121' is already in the middle of another task and cannot be disturbed.

[11/Jan/2017:04:05:48 -0500] - import importest1121: Processed 98924 entries -- average rate 512.6/sec, recent rate 480.0/sec, hit ratio 99%
[11/Jan/2017:04:05:51 -0500] entryrdn-index - _entryrdn_insert_key: Same DN (dn: ou=Netscape Servers,dc=importest,dc=com) is already in the entryrdn file with different ID 160.  Expected ID is 100315.
[11/Jan/2017:04:05:58 -0500] - import importest1121: Duplicated DN detected: "ou=Netscape Servers,dc=importest,dc=com": Entry ID: (100315)
[11/Jan/2017:04:05:59 -0500] - import importest1121: Aborting all Import threads...

[root@cypher 389ds-replica]# date
Wed Jan 11 05:52:52 EST 2017

[root@cypher 389ds-replica]# ldapsearch -LLL -x -p 1121 -h localhost -D "cn=Directory Manager" -w Secret123 -b "dc=importest,dc=com"
Operations error (1)

[root@cypher 389ds-replica]# ldapdelete -x -p 1121 -h localhost -D "cn=Directory Manager" -w Secret123 "cn=import_2017_1_11_4_2_35,cn=import,cn=tasks,cn=config"
ldap_delete: Server is unwilling to perform (53)

[root@cypher 389ds-replica]# rpm -qa |grep -i 389-ds-base
389-ds-base-libs-1.2.11.15-86.el6.x86_64
389-ds-base-debuginfo-1.2.11.15-86.el6.x86_64
389-ds-base-1.2.11.15-86.el6.x86_64
389-ds-base-devel-1.2.11.15-86.el6.x86_64

[root@cypher 389ds-replica]# ldapsearch -x -p 1121 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=import, cn=tasks, cn=config" |grep -i "dn: cn=import_2017_1_11_4_2_35"
dn: cn=import_2017_1_11_4_2_35,cn=import,cn=tasks,cn=config
Comment 12 mreynolds 2017-01-11 17:02:56 UTC 
Hi Sankar, 

This is all expected behavior.  

Since the import fails the database is now hosed.  So the operations error is correct.

Also, you can not delete tasks - the server typically handles that at shutdown.

Mark
Comment 15 Sankar Ramalingam 2017-01-16 09:12:31 UTC 
Based on comment #11 and comment #12, marking the bug as Verified.
Comment 19 errata-xmlrpc 2017-03-21 10:24:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0667.html
Note You need to log in before you can comment on or make changes to this bug.