Bug 1344093
| Summary: | IPA dirsrv start timeout "Detected Disorderly Shutdown last time Directory Server was running, recovering database." | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | lmgnid |
| Component: | 389-ds-base | Assignee: | Noriko Hosoi <nhosoi> |
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Viktor Ashirov <vashirov> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.2 | CC: | lkrispen, lmgnid, nkinder, pvoborni, rcritten, rmeggins, tbordaz |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-04 18:28:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
lmgnid
2016-06-08 18:08:43 UTC
Even I set the timeout to 60000s in /usr/lib/python2.7/site-packages/ipalib/constants.py, the database still cannot be recovered [root@eupreprd-ops-ipa-01 site-packages]# ipactl start --force --ignore-service-check ... ipa: DEBUG: wait_for_open_ports: localhost [389] timeout 60000 If you start DS manually, does it eventually start? If it doesn't then there is no point in increasing ipactl start timeout and the issue in DS must be investigated. # systemctl stop dirsrv the "systemctl stop ..." should be "systemctl start ..." obviously :) Hi Petr, It still cannot be started and the log is the same [root@eupreprd-ops-ipa-01 ~]# /bin/systemctl start 'dirsrv' In log: [10/Jun/2016:16:25:07 +0000] - SSL alert: Configured NSS Ciphers [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_AES_256_GCM_SHA384: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_AES_256_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_CAMELLIA_256_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_AES_128_GCM_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_AES_128_CBC_SHA256: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_CAMELLIA_128_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] - SSL alert: TLS_RSA_WITH_SEED_CBC_SHA: enabled [10/Jun/2016:16:25:07 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2 [10/Jun/2016:16:25:07 +0000] - 389-Directory/1.3.4.0 B2015.322.2137 starting up [10/Jun/2016:16:25:07 +0000] - WARNING: userRoot: entry cache size 10485760B is less than db size 11599872B; We recommend to increase the entry cache size nsslapd-cachememsize. [10/Jun/2016:16:25:07 +0000] - Detected Disorderly Shutdown last time Directory Server was running, recovering database. Adding some 389-ds engineers to cc to get their assistance. There might be more issues with the environment, see bug 1343798 and bug 1343796 Hello, I failed to reproduce the crash. It is reported as systematic so I am likely missing the rigth test case. Did you created a server/replica, with what parameters (DNS, CA...) ? thanks thierry the bug report says always reproducable at VM reboot, but does this mean that the once broken instance does not restart after each reboot or that every time a VM with a functional instance is rebooted the error occurs ? If the dierectory server is not cleanly stopped when the VM is turned off it will have to undergo recovery at the next restart. It should either succeed or log a recovery failure after some time, if this is not the case could you provide a few pstacks of the slapd process to see what it is doing and where it might hang. Recovery is a feature of the used BerkelyDB database backend and debugging might be hard. It would require full access to the __db* database environment files, the transaction log files and the *db database files Hello, thanks for your replys, as I need to recover them urgently, so I don't have the instances with this DB status anymore. But here are some backgrounds and comments: 1: When I try to solve the issue bellow by change the date, basally the AWS ec2 redhat instance will freeze, when I restart the freeze VM, I can see the date was changed back automatically but this DB issue was accrued in 2 IPA servers https://bugzilla.redhat.com/show_bug.cgi?id=1343796 2: As I need to recover the above 2 IPA servers urgently, so one server was recovered by using a AWS ec2 backup, anaother server was recovered by re-installation as in (For eupreprd-ops-ipa-01): https://bugzilla.redhat.com/show_bug.cgi?id=1343798 And some comments for Thierry and Ludwig: - This DB issue happened for already installed replicas with CA and DNS - It always reproducable when VM reboot, or when you start with "ipaclt start", actually the dirsrv can NEVER be started when this issue happens, always timeout after the default 300s - I might be able to provide the DB file or backup, but if I meet this issue again I will provide the necessary logs. Lowering severity according to comment 10 and moving to 389ds. |