Bug 611773

Summary: reindex failure on upgrade
Product: [Retired] 389 Reporter: Alexander Boström <abo>
Component: MigrationAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED WORKSFORME QA Contact: Chandrasekar Kannan <ckannan>
Severity: medium Docs Contact:
Priority: high    
Version: 1.2.6CC: benl, edewata, jgalipea, mastahnke, nhosoi, nkinder, rmeggins
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-09-23 20:33:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 434915, 543590    
Attachments:
Description Flags
error log
none
output from "setup-ds.pl -u" and "setup-ds-admin.pl -u" none

Description Alexander Boström 2010-07-06 13:39:37 UTC
Created attachment 429779 [details]
error log

Description of problem:
After updating to 389-* from epel-testing, all databases are gone.

Version-Release number of selected component (if applicable):
(Updated to these packages)
389-ds-1.2.1-1.el5.noarch.rpm
389-admin-console-doc-1.1.5-1.el5.noarch.rpm
389-ds-console-doc-1.2.3-1.el5.noarch.rpm
389-console-1.1.4-1.el5.noarch.rpm
389-admin-console-1.1.5-1.el5.noarch.rpm
389-admin-1.1.11-0.5.rc1.el5.x86_64.rpm
389-dsgw-1.1.5-1.el5.x86_64.rpm
389-ds-console-1.2.3-1.el5.noarch.rpm
389-ds-base-1.2.6-0.8.rc3.el5.x86_64.rpm

How reproducible:
Always

Steps to Reproduce:
1. CentOS 5 with 389-ds-base-1.2.5-1.el5
2. Three machines with multimaster replication of several databases.
3. service dirsrv stop; service dirsrv-admin stop;
 yum update '389-*' --enablerepo=epel-testing

Actual results:
No databases visible in the graphical console, nothing found in LDAP searches, even though it was there before the upgrade. Errors in the log about "reindex" failure.

Additional info:
I got SELinux errors (SELinux is preventing ns-slapd (dirsrv_t) "create" to <Unknown> (dirsrv_t)), but running the upgrade in permissive mode makes no difference.
These are snapshoted virtual machines so I can restore and retry the process.

Comment 1 Noriko Hosoi 2010-07-08 00:00:18 UTC
Is it possible to share the contents of your DB files?  We'd like to examine them to figure out how the problem was caused in the upgrade.

If you could attach the output of these command lines or send them to us, it'll be a great help.
dbscan -f /var/lib/dirsrv/slapd-<ID>/db/NetscapeRoot/id2entry.db4
dbscan -f /var/lib/dirsrv/slapd-<ID>/db/NetscapeRoot/ancestorid.db4 -r
dbscan -f /var/lib/dirsrv/slapd-<ID>/db/UnixDB/id2entry.db4
Thanks!

Comment 2 Noriko Hosoi 2010-07-08 20:18:16 UTC
I forgot to mention in my previous request.  I'd like to see the output from the database before running the upgrade (389 v1.2.5).  If you happen to keep the back up of the old version, you could run the above command lines against the corresponding back'ed up db files.
Thanks.

Comment 4 Alexander Boström 2010-07-13 15:10:50 UTC
Sure, it's only a test system. Requested data now sent out-of-band.

Comment 6 Noriko Hosoi 2010-07-28 17:13:01 UTC
Created attachment 435082 [details]
output from "setup-ds.pl -u" and "setup-ds-admin.pl -u"

I could not reproduce the problem.

Here's the test steps I took.
1. Installed 389 v1.2.5 with 2 server instances.
   On server A
      o=netscaperoot
      dc=example,dc=com
   On server B
      dc=sub, dc=example,dc=com
1-1. Set up database link (chaining backend) from server A to B on dc=sub,dc=example,dc=com
     Verified that the search with baseDN "dc=sub, dc=example,dc=com" on the server A returns the data
1-2. Set up Single master replication from A to B on o=netscaperoot

2. Installed 389 v1.2.6 (local build from the git trunk)
2-1. setup-ds.pl -u
2-2. setup-ds-admin.pl -u
     The attached file contains the output from the both upgrade scripts.

The recursive ls "ls -R /var/lib/slapd-*/db" shows there are no empty/corrupted db instance dir after the upgrade.

Do you think there could be any other set up/operations I could try to reproduce the problem?

Comment 7 Noriko Hosoi 2010-07-30 21:21:35 UTC
Hello Alexander,

Can we ask you a favour?  We found a bug and the fix is under review:

Bug 619595  - Upgrading sub suffix under non-normalized suffix disappears

We are interested to see if this proposal affects your symptom or not.  If you could reproduce the problem, may we ask you add one more step?

1. Have an older version which contains your original configuration
2. Get 389-ds-base-1.2.6-0.8.rc3.el5.x86_64.rpm from yum repository (e.g., from http://download.fedora.devel.redhat.com/pub/epel/testing/5/x86_64/)
3. Run rpm -U 389-ds-base-1.2.6-0.8.rc3.el5.x86_64.rpm
4. Add this line to /etc/dirsrv/schema/02common.ldif:
attributeTypes: ( 2.16.840.1.113730.3.1.2094 NAME 'nsslapd-parent-suffix' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 X-ORIGIN 'Netscape' )

The diff looks like this
@@ -145,6 +145,7 @@ attributeTypes: ( 1.3.6.1.1.5 NAME 'vendorVersion' EQUALITY 1.3.6.1.4.1.1466.109
 attributeTypes: ( 2.16.840.1.113730.3.1.3023 NAME 'nsViewFilter' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 X-ORIGIN 'Netscape Directory Server' )
 attributeTypes: ( 2.16.840.1.113730.3.1.2063 NAME 'nsEncryptionAlgorithm' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN 'Netscape Directory Server' )
 attributeTypes: ( 2.16.840.1.113730.3.1.2093 NAME 'nsslapd-changelogsuffix' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 X-ORIGIN 'Netscape' )
+attributeTypes: ( 2.16.840.1.113730.3.1.2094 NAME 'nsslapd-parent-suffix' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 X-ORIGIN 'Netscape' )
 #
 # objectclasses:
 #

4. Run setup-ds.pl -u
5. Run setup-ds-admin.pl -u

Then, do yo still observe this problem?
> No databases visible in the graphical console, nothing found in LDAP 
> searches, even though it was there before the upgrade. Errors in the 
> log about "reindex" failure.

Thank you for your help in advance.
--noriko

Comment 8 Rich Megginson 2010-08-02 17:10:24 UTC
The patch has been pushed - shall we mark this bug as MODIFIED?

Comment 9 Noriko Hosoi 2010-08-02 17:25:13 UTC
(In reply to comment #8)
> The patch has been pushed - shall we mark this bug as MODIFIED?    

Well, we are still not sure the cause is identical or not.  (I'm really hoping so, of course... :)  I'm setting "needinfo".

And in the steps in Comment 7, I found a "bug"...  Please start the servers after 4 and before 5.  (Sorry...)

4. Run setup-ds.pl -u

service dirsrv start
^^^^^^^^^^^^^^^^^^^^

5. Run setup-ds-admin.pl -u

Comment 10 Noriko Hosoi 2010-08-11 23:56:43 UTC
The latest release candidate of 389 v1.2.6 contains this bug fix mentioned in Comment 7.  Could it be possible to upgrade your test server to the latest v1.2.6 and see if your problem still occurs?
Thank you for your help, in advance.

Comment 11 Noriko Hosoi 2010-09-23 20:33:15 UTC
Since we have not received the response for the NEEDINFO request for more than a month, we are closing this bug for now.  Please reopen this bug if the problem happens again.