Bug 175170
Summary: | Directory Server Admin Server Dies after Secure Bind to Directory Server | ||
---|---|---|---|
Product: | [Retired] 389 | Reporter: | Ashley Chew <ashley> |
Component: | Admin | Assignee: | Rich Megginson <rmeggins> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Viktor Ashirov <vashirov> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 1.0 | CC: | rcritten |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | i386 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-12-07 16:42:59 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 152373, 183369, 240316 | ||
Attachments: |
Description
Ashley Chew
2005-12-07 08:11:24 UTC
Rob, I think we ran into this one before. Do you remember what the problem was? Oh, wait - permissions? Can you provide the output of ls -l /opt/fedora-ds/alias ? Also, what is the uid that you are using to run the admin server and the directory server? And verify that nss_pcache got started. Thats the permissions of the /opt/fedora-ds/alias [root@naruto alias]# pwd /opt/fedora-ds/alias [root@naruto alias]# ls -al total 404 drwxr-xr-x 2 nobody nobody 4096 Dec 6 14:52 . drwxr-xr-x 17 root root 4096 Dec 6 14:50 .. -rw------- 1 nobody nobody 65536 Dec 6 14:57 admin-serv-naruto-cert8.db -rw------- 1 nobody nobody 24576 Dec 6 14:57 admin-serv-naruto-key3.db -rwxr-xr-x 1 root nobody 194880 Nov 30 06:06 libnssckbi.so -rw------- 1 nobody nobody 16384 Dec 8 09:38 secmod.db -rw------- 1 nobody nobody 65536 Dec 6 14:59 slapd-naruto-cert8.db -rw------- 1 nobody nobody 24576 Dec 6 14:58 slapd-naruto-key3.db Permissions are set for basically for noboy for rw. Now I've looked at the owner process for Fedora Direcotry Server. From the installation script, I choose the default ie Server User ID: nobody Server Group ID: nobody Administration Server: root Now if I look at the process before I switched to bind securely between the Directory Server and Admin Server I see [root@naruto slapd-naruto]# ps -aux | grep -i slap Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ nobody 4240 5.8 4.6 466088 23044 ? Sl 09:51 0:02 ./ns-slapd -D /opt/fedora-ds/slapd-naruto -i /opt/fedora-ds/slapd-naruto/logs/pid -w /opt/fedora-ds/slapd-naruto/logs/startpid root 4318 0.0 0.1 4452 664 pts/2 R+ 09:51 0:00 grep -i slap (slapd) Its running as nobody Now with the Fedora Admin Server before turning on the secure bind between Admin and the DS [root@naruto fedora-ds]# ps -aux | grep -i admin Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ root 4334 0.0 0.3 4772 1688 pts/2 S 09:53 0:00 /opt/fedora-ds/bin/admin/admin/bin/nss_pcache off /opt/fedora-ds/alias admin-serv-naruto- root 4338 0.8 2.1 36828 10800 ? Ssl 09:53 0:00 /usr/sbin//httpd.worker -k start -d /opt/fedora-ds/admin-serv -f /opt/fedora-ds/admin-serv/config/httpd.conf root 4339 0.0 1.2 32316 6212 ? S 09:53 0:00 /usr/sbin//httpd.worker -k start -d /opt/fedora-ds/admin-serv -f /opt/fedora-ds/admin-serv/config/httpd.conf nobody 4342 0.0 2.3 705096 11588 ? Sl 09:53 0:00 /usr/sbin//httpd.worker -k start -d /opt/fedora-ds/admin-serv -f /opt/fedora-ds/admin-serv/config/httpd.conf root 4411 0.0 0.1 3916 688 pts/2 R+ 09:54 0:00 grep -i admin running as root and nobody which is sounds about right. Now I thought about that permission previously about the alias directory. For test purposes I just chmod -R 777 /opt/fedora-ds/alias (Anyone has physical read/write/execute access now) Just to test the permission, then I rebind the Admin Server securely to the Directory Server like I indicated in my initial report and I pretty get much get the same thing ie this error [error] Unable to read from pin store for slot: NSS Certificate DB APR err: 0 With regards to the nss_pcache, thats mean't to be started when you initiate the start-admin or restart-admin script automatically. Without the secure binding between the Admin Server and Directory Server as you see from my process list from above nss_pcache. But as soon as I turned the secure binding between the two, no process which I should see including the nss_pcache from running the admin script ie restart-admin or start-admin show it. When you start-admin in secure mode, does it prompt you for your pin on the command line? I think the answer is yes, inferring from your comments above. Try this: edit start-admin, and make httpd run with strace e.g. strace -o /tmp/httpd.out -f $HTTPD -k start -d $ADMSERV_ROOT -f $ADMSERV_ROOT/config/httpd.conf "$@" as the last line in start-admin. Once you have the file /tmp/httpd.out, please edit it to remove any sensitive information like passwords, then attach it to this bug. Thanks! Created attachment 122023 [details]
strace file from start-admin
I forgot to add, yes you are correct that it does ask me for the password to access the certs for both Admin Server / Directory Server. This automatically happens when you run the restart/start scripts for Admin Server / Directory Server once you entered/created a certificate database key3db/cert8db for either Servers respectively although you may not yet enabled the encryption method if I remembered correctly. Thanks. Rob, if you look at the strace output, the command to pcache is STOR\tNSS Certificate DB\tXXXXXXXX and the error returned is 3, PIN_NOSUCHTOKEN Is "NSS Certificate DB" the correct token name? I saw that too. That is the default token name for an NSS database. It should have been overridden to "internal". I'm not sure how this could have happened. It might be useful to see the file admin-serv/config/console.conf and nss.conf to see how SSL is configured. Created attachment 122053 [details]
Admin Serv Conf file after secure binding betweem Admin and DS
Created attachment 122054 [details]
NSS conf file after DS and ADMIN are rebinded securely
Thanks for those conf files. We need one more - the console.conf file. Created attachment 122055 [details]
Console Configeration File, from Admin Server
Should be attached, the configerations are all from DS / Admin Server after the
attempt of securely binding between DS / Admin Server. I've not directly
modified the files so they should be the default configeration files generated.
That's really odd. In the strace output, it's using "NSS Certificate DB" as the token name, but in your console.conf, it has NSSNickname Server-Cert-ADMIN as the token name. Try this: cd /opt/fedora-ds/alias ../shared/bin/certutil -L -d `pwd` -P admin-serv-naruto- What do you see? You should see a certificate named "Server-Cert-ADMIN". Yes thats correct it should see the Server-Cert-ADMIN which is the Admin Server Cert for my local machine naruto [root@naruto alias]# pwd /opt/fedora-ds/alias [root@naruto alias]# ../shared/bin/certutil -L -d `pwd` -P admin-serv-naruto- Server-Cert-DS u,u,u Server-Cert-ADMIN u,u,u naruto.csse.uwa.edu.au CT,, naruto.csse.uwa.edu.au #2 CT,, [root@naruto alias]# Ok, how about ../shared/bin/modutil -list -dbdir `pwd` -dbprefix admin-serv-naruto- ? I get this. Note that there is a token named "NSS Certificate DB" but the NSS code should alias this to "internal" which is what I think the code should be using. Listing of PKCS #11 Modules ----------------------------------------------------------- 1. NSS Internal PKCS #11 Module slots: 2 slots attached status: loaded slot: NSS Internal Cryptographic Services token: NSS Generic Crypto Services slot: NSS User Private Key and Certificate Services token: NSS Certificate DB 2. Root Certs library name: /opt/fedora-ds/alias/libnssckbi.so slots: 1 slot attached status: loaded slot: token: Builtin Object Token Yeap sure here it is, looks exactly as the same above. [root@naruto alias]# date Fri Dec 9 13:36:34 WST 2005 [root@naruto alias]# pwd /opt/fedora-ds/alias [root@naruto alias]# ../shared/bin/modutil -list -dbdir `pwd` -dbprefix admin-serv-naruto- Using database directory /opt/fedora-ds/alias... Listing of PKCS #11 Modules ----------------------------------------------------------- 1. NSS Internal PKCS #11 Module slots: 2 slots attached status: loaded slot: NSS Internal Cryptographic Services token: NSS Generic Crypto Services slot: NSS User Private Key and Certificate Services token: NSS Certificate DB 2. Root Certs library name: /opt/fedora-ds/alias/libnssckbi.so slots: 1 slot attached status: loaded slot: token: Builtin Object Token ----------------------------------------------------------- Thanks for doing all of this. Unfortunately, we're baffled. There are two parts to this: mod_nss and nss_pcache. nss_pcache is a separate program that caches the pin that you enter on the console when admin server starts up, because due to the way Apache does module initialization, it may need this several times, and you probably don't want to enter it several times. In both places, the module is renamed from "NSS Certificate DB" to "internal" in memory for convenience. However, it appears in your case that in mod_nss, the rename doesn't happen, but in nss_pcache, it does. We can see that because when start-admin prompts you for the password, it's using "NSS Certificate DB" in the prompt rather than "internal". There is a workaround. You can create a pin file for your password. You would need to do this anyway if you wanted remote or unattended restarts of admin server, when there is no console to type the password into. 1) Create a file admin-serv/config/password.conf. This file should contain the following line: NSS Certificate DB:yourpassword where yourpassword is your password. 2) Make sure the file is owned by nobody:nobody and is mode 0400. 3) Edit admin-serv/config/nss.conf. Change the line NSSPassPhraseDialog builtin to NSSPassPhraseDialog file:/opt/fedora-ds/admin-serv/conf/password.conf Then restart-admin. I gave that a shot, seems like it fails as well with the same result (Step 3 small typo error should be /opt/fedora-ds/admin-serv/config/password.conf to be consistent with step 1 just in case if anyones following this thread) [Mon Dec 12 10:03:33 2005] [notice] Access Host filter is: *.csse.uwa.edu.au [Mon Dec 12 10:03:33 2005] [notice] Access Address filter is: * [Mon Dec 12 10:03:33 2005] [error] Unable to read from pin store for slot: NSS Certificate DB APR err: 0 I was testing if it was actualy reading the changes so I substituted a dummy path to see but yes it reading teh changes you suggested or else it would of failed. [root@naruto fedora-ds]# ./restart-admin server not running Syntax error on line 48 of /opt/fedora-ds/admin-serv/config/nss.conf: NSSPassPhraseDialog: file '/opt/fedora-ds/admin-serv/config/password.conf.wrongplace' does not exist Hmmm I thought it would be a trivial thing I guess not. Its just extremely weird that the console can talk to admin serv via securely, ldap command tools can query Directory Server securely but you switch the bindings securely it fails. It dosn't really bother me as I'm using command based ldap tools to do editing and changing but some ppl who are not as fluent using ldap tools would use the GUI admin serv to do the simple things. Cheers then, Ashley We've tested the console with ssl many times - that's why your particular case is so odd. Does it work if you use "internal" instead of "NSS Certificate DB" as the token name in password.conf? Sorry that I took so long, I had other things which I had to do. Anyways I did the changes [root@naruto config]# pwd /opt/fedora-ds/admin-serv/config [root@naruto config]# cat password.conf internal:XXXXXXXX When I make the changes I did it without binding securely the Admin Server / Directory Server although Admin Server and Directory server is running securely. I restart the services for slapd and admin server, slapd comes back up and the admin server comes up. Then I repeat the process now changing the binding between the Admin Server and Directory Server securely. It does the same thing [Thu Dec 15 09:34:01 2005] [notice] caught SIGTERM, shutting down [Thu Dec 15 09:34:07 2005] [notice] Access Host filter is: *.csse.uwa.edu.au [Thu Dec 15 09:34:07 2005] [notice] Access Address filter is: * [Thu Dec 15 09:34:09 2005] [error] Unable to read from pin store for slot: NSS Certificate DB APR err: 0 We've finally been able to duplicate this problem in-house. To cause the problem, under the console open the Admin server, select the Configuration Directory tab and enable Security. When you restart the admin server the problem occurs. It seems to be related to the SSL working being done in mod_admserv. It has its own NSS_Initialize() and since it gets loaded first in the config, that explains why the token wasn't being renamed. Switching the order of the modules in httpd.conf gets the token renamed but mod_nss can't unload because something is holding open a reference so NSS_Shutdown() failes with SEC_ERROR_BUSY. We're investigating. I've found several memory leaks and resource deallocation problems in mod_admserv that may be causing NSS_Shutdown to fail. We're testing a solution. Created attachment 123293 [details]
diffs for mod_admserv.c
This is only part of the solution, which will require changes to mod_nss as
well as the ldap c sdk and admin server.
Created attachment 123294 [details]
diffs for mod_admserv.c
Whoops, wrong diffs
Created attachment 123299 [details]
added some admldapInfo memory leak fixes
Checking in mod_admserv.c; /cvs/dirsec/mod_admserv/mod_admserv.c,v <-- mod_admserv.c new revision: 1.19; previous revision: 1.18 done Reviewed by: Rob C. (Thanks!) Files: mod_admserv.c Branch: HEAD Fix Description: This fix makes the assumption that mod_nss will always be used. It is possible to use mod_admserv without mod_nss - this would mean that the admin server accepts http, but uses ldaps to communicate with the DS. However, I don't forsee that happening, so in order to simplify things, this fix makes mod_nss resposible for initializing NSS and shutting it down properly. Another problem was the memory and resource leaks. pset's have to be disposed of after use. This appears to have been a problem in the old NES libAdmservPlugin as well since most of the code was just copied/pasted. There were also a couple of other memory leaks. NOTE: This is only part of the total fix, which will involve changes to mod_nss, ldap sdk, and admin server components. Platforms tested: FC4 Flag Day: no Doc impact: no |