RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1916677 - A failed re-indexing leaves the database in broken state.
Summary: A failed re-indexing leaves the database in broken state.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: 389-ds-base
Version: 8.3
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: 8.0
Assignee: thierry bordaz
QA Contact: RHDS QE
URL:
Whiteboard: sync-to-jira
Depends On:
Blocks: 1936461
TreeView+ depends on / blocked
 
Reported: 2021-01-15 12:04 UTC by Têko Mihinto
Modified: 2024-03-25 17:51 UTC (History)
7 users (show)

Fixed In Version: 389-ds-1.4-8040020210203193446-866effaa
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1936461 (view as bug list)
Environment:
Last Closed: 2021-05-18 15:45:44 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 4581 0 None closed A failed re-indexing leaves the database in broken state 2021-02-15 12:27:30 UTC
Red Hat Knowledge Base (Solution) 5786981 0 None None None 2021-02-09 01:26:36 UTC

Description Têko Mihinto 2021-01-15 12:04:35 UTC 
Description of problem:
After an offline re-indexing failed, a search on the root entry returned LDAP error 32 ( No such object ).

Version-Release number of selected component (if applicable):
$ cat <SOS_REPORT>/etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)
$
$ grep 389-ds-base installed-rpms
389-ds-base-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64  Mon Nov  9 11:39:09 2020
389-ds-base-libs-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Mon Nov  9 11:39:04 2020
$

How reproducible:
Happened at customer site. Need to check if this is easily reproducible.

Steps to Reproduce:
1. Stop the LDAP server

2. Perform an offline re-indexing
# dsctl slapd-<INSTANCE> db2index <BACKEND>

3. The command returned "db2index failed"

4. Searching the top entry returned error 32:
# ldapsearch -D cn=directory\ manager -W -h <HOST> -p <PORT> -b <ROOT_SUFFIX> -s base
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <ROOT_SUFFIX> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

# search result
search: 2
result: 32 No such object

# numResponses: 1
#

Actual results:
Failed re-indexing and broken database ( corrupt index files? )

Expected results:
Successful re-indexing.

Additional info:
There are almost 8 million of entries and it looks like the re-index job had issues while processing the ancestor IDs:

[07/Jan/2021:16:09:28.994180678 +0100] - INFO - import_monitor_threads - reindex <BACKEND>: Processed 7725746 entries -- average rate 2107.4/sec, recent rate 3174.6/sec, hit ratio 100%
[07/Jan/2021:16:09:50.499630893 +0100] - INFO - import_monitor_threads - reindex <BACKEND>: Workers finished; cleaning up...
[07/Jan/2021:16:09:50.704999955 +0100] - INFO - import_monitor_threads - reindex <BACKEND>: Workers cleaned up.
[07/Jan/2021:16:09:50.707857405 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Cleaning up producer thread...
...
[07/Jan/2021:16:09:50.707857405 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Cleaning up producer thread...
[07/Jan/2021:16:09:50.711312867 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Indexing complete.  Post-processing...
[07/Jan/2021:16:09:50.713715984 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Generating numsubordinates (this may take several minutes to complete)...
[07/Jan/2021:16:09:50.716228418 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Generating numSubordinates complete.
[07/Jan/2021:16:09:50.718969944 +0100] - INFO - bdb_get_nonleaf_ids - reindex <BACKEND>: Gathering ancestorid non-leaf IDs...
[07/Jan/2021:16:22:32.141338774 +0100] - INFO - util_get_hardware_threads - Automatically configuring 16 threads
Comment 15 thierry bordaz 2021-01-29 09:28:19 UTC 
upstream ticket https://github.com/389ds/389-ds-base/issues/4581
Comment 16 thierry bordaz 2021-02-01 12:53:18 UTC 
Fix pushed upstream -> POST
Comment 20 Têko Mihinto 2021-02-05 17:33:45 UTC 
Hi Thierry,

The customer confirmed that the patch fixes the re-indexing issue.

Thank you!

Regards,
Têko.
Comment 21 Akshay Adhikari 2021-02-12 09:19:18 UTC 
Build Tested: 389-ds-base-1.4.3.16-10.module+el8.4.0+9902+6195a65d.x86_64

The patch is available in the source, also the import test-suite is passing.

============================================================================ test session starts ================================================================
platform linux -- Python 3.6.8, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3.6
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-4.18.0-262.el8.x86_64-x86_64-with-redhat-8.4-Ootpa', 'Packages': {'pytest': '6.2.1', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.11.0', 'html': '3.1.1'}}
389-ds-base: 1.4.3.16-10.module+el8.4.0+9902+6195a65d
nss: 3.53.1-11.el8_2
nspr: 4.25.0-2.el8_2
openldap: 2.4.46-16.el8
cyrus-sasl: not installed
FIPS: disabled
rootdir: /root/389-ds-base/dirsrvtests, configfile: pytest.ini
plugins: metadata-1.11.0, html-3.1.1
collected 14 items                                                                                                                                                          

dirsrvtests/tests/suites/import/import_test.py::test_import_with_index PASSED                                                                                         [  7%]
dirsrvtests/tests/suites/import/import_test.py::test_online_import_with_warning PASSED                                                                                [ 14%]
dirsrvtests/tests/suites/import/import_test.py::test_crash_on_ldif2db PASSED                                                                                          [ 21%]
dirsrvtests/tests/suites/import/import_test.py::test_ldif2db_allows_entries_without_a_parent_to_be_imported PASSED                                                    [ 28%]
dirsrvtests/tests/suites/import/import_test.py::test_ldif2db_syntax_check PASSED                                                                                      [ 35%]
dirsrvtests/tests/suites/import/import_test.py::test_issue_a_warning_if_the_cache_size_is_smaller PASSED                                                              [ 42%]
dirsrvtests/tests/suites/import/import_test.py::test_fast_slow_import PASSED                                                                                          [ 50%]
dirsrvtests/tests/suites/import/import_test.py::test_entry_with_escaped_characters_fails_to_import_and_index PASSED                                                   [ 57%]
dirsrvtests/tests/suites/import/import_test.py::test_import_perf_after_failure PASSED                                                                                 [ 64%]
dirsrvtests/tests/suites/import/regression_test.py::test_replay_import_operation PASSED                                                                               [ 71%]
dirsrvtests/tests/suites/import/regression_test.py::test_import_be_default PASSED                                                                                     [ 78%]
dirsrvtests/tests/suites/import/regression_test.py::test_del_suffix_import PASSED                                                                                     [ 85%]
dirsrvtests/tests/suites/import/regression_test.py::test_del_suffix_backend PASSED                                                                                    [ 92%]
dirsrvtests/tests/suites/import/regression_test.py::test_import_duplicate_dn PASSED                                                                                   [100%]

================================================================ 14 passed, 41 warnings in 192.30s (0:03:12) ====================================================

-> Marking as verified: tested.
Comment 24 sgouvern 2021-02-19 08:35:11 UTC 
As per comment 21, marking as VERIFIED
Comment 29 errata-xmlrpc 2021-05-18 15:45:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (389-ds:1.4 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1835
Comment 30 Red Hat Bugzilla 2023-09-15 00:58:22 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days
Note You need to log in before you can comment on or make changes to this bug.