Does ldapsearch work from the command line? e.g.

ldapsearch -x -D "uid=admin, ou=Administrators, ou=TopologyManagement, o=NetscapeRoot" -w "yourpassword" -s base -b ""

Nope:

ldap_bind: Invalid credentials (49)

But it does work with admin 1.1.6 and ds 1.1.2 which is what I'm running in production.

I'm testing by copying over /etc/dirsrv and /var/lib/dirsrv from my production machine to a test machine and upgrading the software there.  Is it possible I'm not copying a needed file over?  System seems to work fine before the upgrade though.

Note that I'm also updating ds-base from 1.1.2 -> 1.2.4 as part of this upgrade.

dn:
objectClass: top
namingContexts: dc=nwra, dc=com
namingContexts: o=netscaperoot
supportedExtension: 2.16.840.1.113730.3.5.7
supportedExtension: 2.16.840.1.113730.3.5.8
supportedExtension: 2.16.840.1.113730.3.5.3
supportedExtension: 2.16.840.1.113730.3.5.5
supportedExtension: 2.16.840.1.113730.3.5.6
supportedExtension: 2.16.840.1.113730.3.5.9
supportedExtension: 2.16.840.1.113730.3.5.4
supportedExtension: 1.3.6.1.4.1.1466.20037
supportedExtension: 1.3.6.1.4.1.4203.1.11.1
supportedControl: 2.16.840.1.113730.3.4.2
supportedControl: 2.16.840.1.113730.3.4.3
supportedControl: 2.16.840.1.113730.3.4.4
supportedControl: 2.16.840.1.113730.3.4.5
supportedControl: 1.2.840.113556.1.4.473
supportedControl: 2.16.840.1.113730.3.4.9
supportedControl: 2.16.840.1.113730.3.4.16
supportedControl: 2.16.840.1.113730.3.4.15
supportedControl: 2.16.840.1.113730.3.4.17
supportedControl: 2.16.840.1.113730.3.4.19
supportedControl: 1.3.6.1.4.1.42.2.27.8.5.1
supportedControl: 1.3.6.1.4.1.42.2.27.9.5.2
supportedControl: 2.16.840.1.113730.3.4.14
supportedControl: 2.16.840.1.113730.3.4.20
supportedControl: 1.3.6.1.4.1.1466.29539.12
supportedControl: 2.16.840.1.113730.3.4.12
supportedControl: 2.16.840.1.113730.3.4.18
supportedControl: 2.16.840.1.113730.3.4.13
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: LOGIN
supportedSASLMechanisms: DIGEST-MD5
supportedSASLMechanisms: PLAIN
supportedSASLMechanisms: ANONYMOUS
supportedSASLMechanisms: CRAM-MD5
supportedSASLMechanisms: GSSAPI
supportedLDAPVersion: 2
supportedLDAPVersion: 3
vendorName: Fedora Project
vendorVersion: Fedora-Directory/1.1.2 B2008.259.1552
dataversion: 020091223000051020091223000051
netscapemdsuffix: cn=ldap://dc=ldap,dc=cora,dc=nwra,dc=com:389

I can still bind to the serve with "cn=Directory Manager".

I still have the entry, and it is exactly the same as on the production server:

# admin, Administrators, TopologyManagement, NetscapeRoot
dn: uid=admin, ou=Administrators, ou=TopologyManagement, o=NetscapeRoot
givenName: Configuration
objectClass: top
objectClass: person
objectClass: organizationalperson
objectClass: inetorgperson
uid: admin
cn: Configuration Administrator
sn: Administrator
userPassword:: e1XXXXXXXXXXXXXXXXXXXXX==

(Omitted the password)

Can you verify what version of 389-ds-base you are using?
This sounds suspiciously like https://bugzilla.redhat.com/show_bug.cgi?id=518520 which was fixed in 389-ds-base 1.2.2

What encoding does the password use?  The userPassword is base64 encoded - you can see the actual value by using python:
python
> import base64
> base64.b64decode('the value')

Is the password using {SHA} or {SSHA} or something else?

It's using {SHA}.

389-ds-base-1.2.4-1.el5

ok - try this:

python
> import base64
> import hashlib
> sha = hashlib.sha1('your password')
> base64.b64encode(sha.digest())

Does the value it prints out match your password (that is, the stuff that comes after the {SHA} in your password)?

Running through the debugger, it's getting called with:

Breakpoint 1, sha_pw_cmp (userpwd=0xa118918 "XXXXXXXX", 
    dbpwd=0xa181f95 "XXXXXXXXXXXXXXXXX", shaLen=20)
    at ldap/servers/plugins/pwdstorage/sha_pwd.c:72             

Resulting in the error:

$10 = 0x16ac80 "pw_cmp: %s userPassword \"%s\" is the wrong length or is not properly encoded BASE64\n"

Which isn't getting logged because no debugging is turned on.  How can I turn on debugging so I can see these messages?

Looks like shaLen is hardcoded to 20?  I assume it should match the length of dbpwd?

I've obfuscated the passwords, but dbpwd is 29 characters, and the last character is a newline, which is strange, but perhaps valid in the hash.

(gdb) print dbpwd[28]
$14 = 10 '\n'
(gdb) print dbpwd[29]
$15 = 0 '\0'

(In reply to comment #7)
> Does the value it prints out match your password (that is, the stuff that comes
> after the {SHA} in your password)?  


It does, minus the newline character.

> How can I turn on debugging so I can see these messages?

These messages are logged at the Plug-in debugging log level - see http://directory.fedoraproject.org/wiki/FAQ#Troubleshooting

> Looks like shaLen is hardcoded to 20?

Right.  The SHA algorithm always produces 20 bytes.

> I assume it should match the length of dbpwd?

Yes and no.  dbpwd is the base64 encoded sha hash of the password.  If you base64 encode 20 bytes, you will get 28 bytes.

In python:
>>> len(sha.digest())
20
>>> len(base64.b64encode(sha.digest()))
28

So dbpwd should be 28 bytes.  I think the problem is that the password should not contain the trailing newline.  Do you remember what version of fedora-ds-base and fedora-ds-admin were used to create the initial password?

(In reply to comment #10)
> So dbpwd should be 28 bytes.  I think the problem is that the password should
> not contain the trailing newline.  Do you remember what version of
> fedora-ds-base and fedora-ds-admin were used to create the initial password?  

Quite likely fedora-ds-base-1.1.0 and fedora-ds-admin-1.1.2.  I'm currently running base-1.1.2 and admin-1.1.6.

I removed the trailing 'Cg==' from the userPassword which corresponded to the newline character and now I can authenticate as the admin user.  I made the change on the production server and can still authenticate there as well.

I don't know if it's worth it or possible to try to fix this automatically during upgrades, of it I'm the only one who ended up with a newline encoded in the password somehow.

Thanks for all the help.

Thanks for confirming my suspicions about the newline.  I think this was a bug that was fixed quite some time ago, that it didn't trim the trailing newline when creating the password.  I'd rather not attempt to fix the userPassword value.  I think the solution that will cause the least disruption will be to change the sha_pw_cmp routine to ignore a trailing newline.

I'm a bit confused here - you say https://bugzilla.redhat.com/show_bug.cgi?id=552421#c8

I've obfuscated the passwords, but dbpwd is 29 characters, and the last
character is a newline, which is strange, but perhaps valid in the hash.

(gdb) print dbpwd[28]
$14 = 10 '\n'
(gdb) print dbpwd[29]
$15 = 0 '\0'

dbpwd is the base64 encoded password - apparently the password was hashed, then encoded, then the \n stuck on the end, then stored to the database.

But later you say https://bugzilla.redhat.com/show_bug.cgi?id=552421#c11

I removed the trailing 'Cg==' from the userPassword which corresponded to the
newline character and now I can authenticate as the admin user.

So the dbpwd looked like this?

dbpwd[24] == 'C'
dbpwd[25] == 'g'
dbpwd[26] == '='
dbpwd[27] == '='
dbpwd[28] == '\n'
dbpwd[29] == '\0'

If this is the case, it looks like this is what happened:
The password was hashed, then a newline was added, then it was base64 encoded, then another newline was added.

Your last summary is correct, but dbpwd is somewhat confused.

userPassword:: in the ldap database was something like:

'e1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXCg=='

decoded it became:

'{SHA}XXXXXXXXXXXXXXXXXXXXXXXXXX0=\n'

with the newline character there.

dbpwd[26] == '0'
dbpwd[27] == '='
dbpwd[28] == '\n'
dbpwd[29] == '\0'

>>> base64.b64encode('\n')
'Cg=='

Ok - I thought you meant dbpwd inside the debugger.  The variable in sha_pw_cmp() used to hold the database password is dbpwd.  It's confusing because ldapsearch always base64 encodes the userPassword attribute, even though it's already base64 encoded.

So the problem is that the password was hashed then added to the database like this:
userPassword: "{SHA}" + <b64encoded(hash)> + "\n"

Created attachment 383080 [details]
patch

To ssh://git.fedorahosted.org/git/389/ds.git
   786c740..2521eb7  master -> master

commit 2521eb7a649fab115366a652d2ec499ad205ae03
Author: Rich Megginson <rmeggins>
Date:   Mon Jan 11 11:51:39 2010 -0700

    Workaround bogus base64 encoded passwords that end in newline
    
    https://bugzilla.redhat.com/show_bug.cgi?id=552421
    Resolves: bug 552421
    Bug Description: Cannot log into admin server after upgrade (fedora-ds-admin
    Reviewed by: nkinder (Thanks!)
    Branch: HEAD
    Fix Description: Some older versions of setup encoded the admin password in 
    newline character in the dbpwd.  newline is not a valid base64 character.
    Platforms tested: RHEL5 x86_64
    Flag Day: no
    Doc impact: no

This is working for me, though I think I've fixed my database so I'm not a good tester anymore.

Hi Rich,

Request you to please guide me on how should I reproduce it with RHDS.

Thanks,
Amita

(In reply to comment #19)
> Hi Rich,
> 
> Request you to please guide me on how should I reproduce it with RHDS.
> 
> Thanks,
> Amita

There is nothing to verify with 389 at this time.  We will test this as part of rhds to 389 migration testing.

Amita, since you were able to upgrade from 8.2 to 9.0 successfully, you can mark this as VERIFIED.

ok, marking as VERIFIED.