Bug 1916677

Summary: A failed re-indexing leaves the database in broken state.
Product: Red Hat Enterprise Linux 8 Reporter: Têko Mihinto <tmihinto>
Component: 389-ds-baseAssignee: thierry bordaz <tbordaz>
Status: CLOSED ERRATA QA Contact: RHDS QE <ds-qe-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.3CC: aadhikar, ldap-maint, mreynolds, msauton, sgouvern, tbordaz, toneata
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: sync-to-jira
Fixed In Version: 389-ds-1.4-8040020210203193446-866effaa Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1936461 (view as bug list) Environment:
Last Closed: 2021-05-18 15:45:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1936461    

Description Têko Mihinto 2021-01-15 12:04:35 UTC
Description of problem:
After an offline re-indexing failed, a search on the root entry returned LDAP error 32 ( No such object ).

Version-Release number of selected component (if applicable):
$ cat <SOS_REPORT>/etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)
$
$ grep 389-ds-base installed-rpms
389-ds-base-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64  Mon Nov  9 11:39:09 2020
389-ds-base-libs-1.4.3.13-1.module+el8dsrv+8334+69a46a2e.x86_64 Mon Nov  9 11:39:04 2020
$

How reproducible:
Happened at customer site. Need to check if this is easily reproducible.

Steps to Reproduce:
1. Stop the LDAP server

2. Perform an offline re-indexing
# dsctl slapd-<INSTANCE> db2index <BACKEND>

3. The command returned "db2index failed"

4. Searching the top entry returned error 32:
# ldapsearch -D cn=directory\ manager -W -h <HOST> -p <PORT> -b <ROOT_SUFFIX> -s base
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <ROOT_SUFFIX> with scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

# search result
search: 2
result: 32 No such object

# numResponses: 1
#

Actual results:
Failed re-indexing and broken database ( corrupt index files? )

Expected results:
Successful re-indexing.

Additional info:
There are almost 8 million of entries and it looks like the re-index job had issues while processing the ancestor IDs:

[07/Jan/2021:16:09:28.994180678 +0100] - INFO - import_monitor_threads - reindex <BACKEND>: Processed 7725746 entries -- average rate 2107.4/sec, recent rate 3174.6/sec, hit ratio 100%
[07/Jan/2021:16:09:50.499630893 +0100] - INFO - import_monitor_threads - reindex <BACKEND>: Workers finished; cleaning up...
[07/Jan/2021:16:09:50.704999955 +0100] - INFO - import_monitor_threads - reindex <BACKEND>: Workers cleaned up.
[07/Jan/2021:16:09:50.707857405 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Cleaning up producer thread...
...
[07/Jan/2021:16:09:50.707857405 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Cleaning up producer thread...
[07/Jan/2021:16:09:50.711312867 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Indexing complete.  Post-processing...
[07/Jan/2021:16:09:50.713715984 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Generating numsubordinates (this may take several minutes to complete)...
[07/Jan/2021:16:09:50.716228418 +0100] - INFO - bdb_import_main - reindex <BACKEND>: Generating numSubordinates complete.
[07/Jan/2021:16:09:50.718969944 +0100] - INFO - bdb_get_nonleaf_ids - reindex <BACKEND>: Gathering ancestorid non-leaf IDs...
[07/Jan/2021:16:22:32.141338774 +0100] - INFO - util_get_hardware_threads - Automatically configuring 16 threads

Comment 15 thierry bordaz 2021-01-29 09:28:19 UTC
upstream ticket https://github.com/389ds/389-ds-base/issues/4581

Comment 16 thierry bordaz 2021-02-01 12:53:18 UTC
Fix pushed upstream -> POST

Comment 20 Têko Mihinto 2021-02-05 17:33:45 UTC
Hi Thierry,

The customer confirmed that the patch fixes the re-indexing issue.

Thank you!

Regards,
Têko.

Comment 21 Akshay Adhikari 2021-02-12 09:19:18 UTC
Build Tested: 389-ds-base-1.4.3.16-10.module+el8.4.0+9902+6195a65d.x86_64

The patch is available in the source, also the import test-suite is passing.

============================================================================ test session starts ================================================================
platform linux -- Python 3.6.8, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3.6
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-4.18.0-262.el8.x86_64-x86_64-with-redhat-8.4-Ootpa', 'Packages': {'pytest': '6.2.1', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.11.0', 'html': '3.1.1'}}
389-ds-base: 1.4.3.16-10.module+el8.4.0+9902+6195a65d
nss: 3.53.1-11.el8_2
nspr: 4.25.0-2.el8_2
openldap: 2.4.46-16.el8
cyrus-sasl: not installed
FIPS: disabled
rootdir: /root/389-ds-base/dirsrvtests, configfile: pytest.ini
plugins: metadata-1.11.0, html-3.1.1
collected 14 items                                                                                                                                                          

dirsrvtests/tests/suites/import/import_test.py::test_import_with_index PASSED                                                                                         [  7%]
dirsrvtests/tests/suites/import/import_test.py::test_online_import_with_warning PASSED                                                                                [ 14%]
dirsrvtests/tests/suites/import/import_test.py::test_crash_on_ldif2db PASSED                                                                                          [ 21%]
dirsrvtests/tests/suites/import/import_test.py::test_ldif2db_allows_entries_without_a_parent_to_be_imported PASSED                                                    [ 28%]
dirsrvtests/tests/suites/import/import_test.py::test_ldif2db_syntax_check PASSED                                                                                      [ 35%]
dirsrvtests/tests/suites/import/import_test.py::test_issue_a_warning_if_the_cache_size_is_smaller PASSED                                                              [ 42%]
dirsrvtests/tests/suites/import/import_test.py::test_fast_slow_import PASSED                                                                                          [ 50%]
dirsrvtests/tests/suites/import/import_test.py::test_entry_with_escaped_characters_fails_to_import_and_index PASSED                                                   [ 57%]
dirsrvtests/tests/suites/import/import_test.py::test_import_perf_after_failure PASSED                                                                                 [ 64%]
dirsrvtests/tests/suites/import/regression_test.py::test_replay_import_operation PASSED                                                                               [ 71%]
dirsrvtests/tests/suites/import/regression_test.py::test_import_be_default PASSED                                                                                     [ 78%]
dirsrvtests/tests/suites/import/regression_test.py::test_del_suffix_import PASSED                                                                                     [ 85%]
dirsrvtests/tests/suites/import/regression_test.py::test_del_suffix_backend PASSED                                                                                    [ 92%]
dirsrvtests/tests/suites/import/regression_test.py::test_import_duplicate_dn PASSED                                                                                   [100%]

================================================================ 14 passed, 41 warnings in 192.30s (0:03:12) ====================================================

-> Marking as verified: tested.

Comment 24 sgouvern 2021-02-19 08:35:11 UTC
As per comment 21, marking as VERIFIED

Comment 29 errata-xmlrpc 2021-05-18 15:45:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (389-ds:1.4 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1835

Comment 30 Red Hat Bugzilla 2023-09-15 00:58:22 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days