Bug 1254412 - when dirsrv is off ,upgrade from 7.1 to 7.2 fails with starting CA and named-pkcs11.service
when dirsrv is off ,upgrade from 7.1 to 7.2 fails with starting CA and named-...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa (Show other bugs)
7.2
Unspecified Unspecified
medium Severity unspecified
: rc
: ---
Assigned To: IPA Maintainers
Namita Soman
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-17 23:44 EDT by Xiyang Dong
Modified: 2015-11-30 09:24 EST (History)
6 users (show)

See Also:
Fixed In Version: ipa-4.2.0-5.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-19 07:05:37 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
389-ds log (50.11 KB, text/plain)
2015-08-17 23:44 EDT, Xiyang Dong
no flags Details
ipaupgrade.log (3.72 MB, text/plain)
2015-08-18 00:10 EDT, Xiyang Dong
no flags Details

  None (edit)
Description Xiyang Dong 2015-08-17 23:44:32 EDT
Created attachment 1064146 [details]
389-ds log

Description of problem:
when dirsrv is off ,upgrade from 7.1 to 7.2 fails with starting CA and named-pkcs11.service

Version-Release number of selected component (if applicable):
ipa-server-4.1.0-18.el7.x86_64 -> ipa-server-4.2.0-4.el7.x86_64
pki-ca-10.1.2-7.el7.noarch -> pki-ca-10.2.5-5.el7.noarch
389-ds-base-1.3.3.1-13.el7.x86_64 -> 389-ds-base-1.3.4.0-11.el7.x86_64
bind-pkcs11-9.9.4-28.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. 7.1 server installed
2. stop dirsrv
3. upgrade to 7.2

Actual results:
upgrade from 7.1 to 7.2 fails with starting CA and named-pkcs11.service

Expected results:
Upgrade success with no failures.

Additional info:

[root@cloud-qe-3 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.1 (Maipo)

[root@cloud-qe-3 ~]# systemctl stop dirsrv.target
[root@cloud-qe-3 ~]# systemctl status dirsrv.target
dirsrv.target - 389 Directory Server
   Loaded: loaded (/usr/lib/systemd/system/dirsrv.target; disabled)
   Active: inactive (dead)
[root@cloud-qe-3 ~]# yum -y update 'ipa*' sssd
.
.
.
  Cleanup    : systemd-libs-208-20.el7.x86_64                                                      134/136 
  Cleanup    : libsss_idmap-1.12.2-58.el7.x86_64                                                   135/136 
  Cleanup    : slapi-nis-0.54-2.el7.x86_64                                                         136/136 
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
CA did not start in 300.0s
MYNEWREPO1/productid                                                                | 1.6 kB  00:00:00     
  Verifying  : 32:bind-libs-lite-9.9.4-28.el7.x86_64                                                 1/136 
  Verifying  : 32:bind-utils-9.9.4-28.el7.x86_64                                                     2/136 
  Verifying  : 389-ds-base-libs-1.3.4.0-11.el7.x86_64                                                3/136 
  Verifying  : pki-server-10.2.5-5.el7.noarch                                                        4/136 
  Verifying  : systemd-python-219-11.el7.x86_64                                                      5/136 
.
.
.


[root@cloud-qe-3 ~]# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services
ipa: INFO: The ipactl command was successful
[root@cloud-qe-3 ~]# ipactl restart
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Job for named-pkcs11.service failed because the control process exited with error code. See "systemctl status named-pkcs11.service" and "journalctl -xe" for details.
Failed to start named Service
Shutting down
Aborting ipactl

[root@cloud-qe-3 ~]# systemctl status named-pkcs11 -l
● named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native PKCS#11
   Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2015-08-17 23:05:29 EDT; 34min ago
  Process: 19865 ExecStart=/usr/sbin/named-pkcs11 -u named $OPTIONS (code=exited, status=1/FAILURE)
  Process: 19862 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS)

Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com named-pkcs11[19867]: adjusted limit on open files from 4096 to 1048576
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com named-pkcs11[19867]: found 4 CPUs, using 4 worker threads
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com named-pkcs11[19867]: using 4 UDP listeners per interface
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com named-pkcs11[19867]: using up to 4096 sockets
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com named-pkcs11[19867]: ObjectStore.cpp(59): Failed to enumerate object store in /var/lib/softhsm/tokens/
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com named-pkcs11[19867]: SoftHSM.cpp(456): Could not load the object store
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com systemd[1]: named-pkcs11.service: control process exited, code=exited status=1
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com systemd[1]: Unit named-pkcs11.service entered failed state.
Aug 17 23:05:29 cloud-qe-3.idmqe.lab.eng.bos.redhat.com systemd[1]: named-pkcs11.service failed.
Comment 2 Xiyang Dong 2015-08-18 00:10:26 EDT
Created attachment 1064148 [details]
ipaupgrade.log
Comment 4 Petr Vobornik 2015-08-18 04:12:32 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/5232
Comment 7 Xiyang Dong 2015-08-31 23:18:17 EDT
Verified on ipa-server.x86_64 0:4.2.0-8.el7:

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: [   LOG    ] :: ipa_upgrade_master_replica_client_dirsrv_off_1: test with dirsrv off before upgrade with new master, old replica, and old client 
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

:: [ 16:51:03 ] :: Shutting down dirsrv before upgrading MASTER (cloud-qe-19.testrelm.test)
:: [  BEGIN   ] :: Running 'systemctl stop dirsrv.target'
:: [   PASS   ] :: Command 'systemctl stop dirsrv.target' (Expected 0, got 0)
:: [ 16:51:03 ] :: upgrade_master: upgrade ipa master
.
.
.
:: [  BEGIN   ] :: Running 'yum -y update 'ipa*' sssd'
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Resolving Dependencies
--> Running transaction check---> Package ipa-admintools.x86_64 0:4.1.0-18.el7_1.3 will be updated
---> Package ipa-admintools.x86_64 0:4.2.0-8.el7 will be an update
---> Package ipa-client.x86_64 0:4.1.0-18.el7_1.3 will be updated
---> Package ipa-client.x86_64 0:4.2.0-8.el7 will be an update
--> Processing Dependency: certmonger >= 0.78 for package: ipa-client-4.2.0-8.el7.x86_64
---> Package ipa-python.x86_64 0:4.1.0-18.el7_1.3 will be updated
---> Package ipa-python.x86_64 0:4.2.0-8.el7 will be an update
--> Processing Dependency: python-yubico >= 1.2.3 for package: ipa-python-4.2.0-8.el7.x86_64
--> Processing Dependency: python-cryptography for package: ipa-python-4.2.0-8.el7.x86_64
--> Processing Dependency: python-libipa_hbac for package: ipa-python-4.2.0-8.el7.x86_64
--> Processing Dependency: python-sss-murmur for package: ipa-python-4.2.0-8.el7.x86_64
---> Package ipa-server.x86_64 0:4.1.0-18.el7_1.3 will be obsoleted
---> Package ipa-server.x86_64 0:4.1.0-18.el7_1.3 will be updated
---> Package ipa-server.x86_64 0:4.2.0-8.el7 will be obsoleting
.
.
.
Installed:
  ipa-server.x86_64 0:4.2.0-8.el7           ipa-server-dns.x86_64 0:4.2.0-8.el7
  python-libipa_hbac.x86_64 0:1.13.0-25.el7

Dependency Installed:
  bind-pkcs11.x86_64 32:9.9.4-28.el7                                            
  bind-pkcs11-libs.x86_64 32:9.9.4-28.el7                                       
  bind-pkcs11-utils.x86_64 32:9.9.4-28.el7                                      
  jackson.noarch 0:1.9.4-7.el7                                                  
  joda-convert.noarch 0:1.3-5.el7                                               
  joda-time.noarch 0:2.2-3.tzdata2013c.el7                                      
  jsr-311.noarch 0:1.1.1-6.el7                                                  
  ldns.x86_64 0:1.6.16-7.el7                                                    
  mod_auth_gssapi.x86_64 0:1.2.0-1.el7                                          
  nuxwdog.x86_64 0:1.0.3-2.el7                                                  
  nuxwdog-client-java.x86_64 0:1.0.3-2.el7                                      
  objectweb-asm.noarch 0:3.3.1-9.el7                                            
  opencryptoki.x86_64 0:3.2-4.1.el7                                             
  opencryptoki-libs.x86_64 0:3.2-4.1.el7                                        
  opencryptoki-swtok.x86_64 0:3.2-4.1.el7                                       
  opendnssec.x86_64 0:1.4.7-2.el7                                               
  pki-kra.noarch 0:10.2.5-5.el7                                                 
  python-cffi.x86_64 0:0.8.6-2.el7                                              
  python-cryptography.x86_64 0:0.8.2-1.el7                                      
  python-enum34.noarch 0:1.0.4-1.el7                                            
  python-kdcproxy.noarch 0:0.3.2-1.el7                                          
  python-ply.noarch 0:3.4-10.el7                                                
  python-pycparser.noarch 0:2.14-1.el7                                          
  python-sss-murmur.x86_64 0:1.13.0-25.el7                                      
  resteasy-base-client.noarch 0:3.0.6-1.el7                                     
  resteasy-base-jackson-provider.noarch 0:3.0.6-1.el7                           
  samba-client-libs.x86_64 0:4.2.3-6.el7                                        
  softhsm.x86_64 0:2.0.0rc1-3.el7                                               

Updated:
  dracut.x86_64 0:033-328.el7          ipa-admintools.x86_64 0:4.2.0-8.el7     
  ipa-client.x86_64 0:4.2.0-8.el7      ipa-python.x86_64 0:4.2.0-8.el7         
  sssd.x86_64 0:1.13.0-25.el7         

Dependency Updated:
  389-ds-base.x86_64 0:1.3.4.0-14.el7                                           
  389-ds-base-libs.x86_64 0:1.3.4.0-14.el7                                      
  bind.x86_64 32:9.9.4-28.el7                                                   
  bind-dyndb-ldap.x86_64 0:8.0-1.el7                                            
  bind-libs.x86_64 32:9.9.4-28.el7                                              
  bind-libs-lite.x86_64 32:9.9.4-28.el7                                         
  bind-license.noarch 32:9.9.4-28.el7                                           
  bind-utils.x86_64 32:9.9.4-28.el7                                             
  certmonger.x86_64 0:0.78.4-1.el7                                              
  dracut-config-rescue.x86_64 0:033-328.el7                                     
  dracut-network.x86_64 0:033-328.el7                                           
  kmod.x86_64 0:20-5.el7                                                        
  krb5-libs.x86_64 0:1.13.2-9.el7                                               
  krb5-pkinit.x86_64 0:1.13.2-9.el7                                             
  krb5-server.x86_64 0:1.13.2-9.el7                                             
  krb5-workstation.x86_64 0:1.13.2-9.el7                                        
  libgudev1.x86_64 0:219-11.el7                                                 
  libipa_hbac.x86_64 0:1.13.0-25.el7                                            
  libsmbclient.x86_64 0:4.2.3-6.el7                                             
  libsss_idmap.x86_64 0:1.13.0-25.el7                                           
  libwbclient.x86_64 0:4.2.3-6.el7                                              
  pki-base.noarch 0:10.2.5-5.el7                                                
  pki-ca.noarch 0:10.2.5-5.el7                                                  
  pki-server.noarch 0:10.2.5-5.el7                                              
  pki-tools.x86_64 0:10.2.5-5.el7                                               
  python-six.noarch 0:1.9.0-2.el7                                               
  python-sssdconfig.noarch 0:1.13.0-25.el7                                      
  python-yubico.noarch 0:1.2.3-1.el7                                            
  samba-common.noarch 0:4.2.3-6.el7                                             
  samba-libs.x86_64 0:4.2.3-6.el7                                               
  selinux-policy.noarch 0:3.13.1-46.el7                                         
  selinux-policy-targeted.noarch 0:3.13.1-46.el7                                
  sssd-ad.x86_64 0:1.13.0-25.el7                                                
  sssd-client.x86_64 0:1.13.0-25.el7                                            
  sssd-common.x86_64 0:1.13.0-25.el7                                            
  sssd-common-pac.x86_64 0:1.13.0-25.el7                                        
  sssd-ipa.x86_64 0:1.13.0-25.el7                                               
  sssd-krb5.x86_64 0:1.13.0-25.el7                                              
  sssd-krb5-common.x86_64 0:1.13.0-25.el7                                       
  sssd-ldap.x86_64 0:1.13.0-25.el7                                              
  sssd-proxy.x86_64 0:1.13.0-25.el7                                             
  systemd.x86_64 0:219-11.el7                                                   
  systemd-libs.x86_64 0:219-11.el7                                              
  systemd-python.x86_64 0:219-11.el7                                            
  systemd-sysv.x86_64 0:219-11.el7                                              
  tomcatjss.noarch 0:7.1.2-1.el7                                                

Replaced:
  ipa-server.x86_64 0:4.1.0-18.el7_1.3                                          
  libipa_hbac-python.x86_64 0:1.12.2-58.el7_1.6                                 

Complete!
:: [   PASS   ] :: Command 'yum -y update 'ipa*' sssd' (Expected 0, got 0)
Comment 8 errata-xmlrpc 2015-11-19 07:05:37 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2362.html
Comment 9 David Dimoski 2015-11-30 09:24:32 EST
Hello,
the above described problem occurs for us after after upgrade from: redhat-release-server-7.1-1 to redhat-release-server-7.2-9 and from ipa-server-4.1.0-18 to ipa-server-4.2.0-15.


[15:21:31 INFRA root@ipa-2 ~]# systemctl status named-pkcs11.service                                                                                                                          
â named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native PKCS#11                                                                                                              
   Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled; vendor preset: disabled)                                                                                           
   Active: failed (Result: exit-code) since Mon 2015-11-30 15:04:57 CET; 17min ago                                                                                                            
  Process: 15658 ExecStart=/usr/sbin/named-pkcs11 -u named $OPTIONS (code=exited, status=1/FAILURE)                                                                                           
  Process: 15655 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z /etc/named.conf; else echo "Checking of zone files is disabled"; fi (code=exited, status=0/SUCCESS)                                                                                                                                                                
                                                                                                                                                                                              
Nov 30 15:04:57 ipa-2.mgmt.hss.int named-pkcs11[15661]: found 4 CPUs, using 4 worker threads                                                                                                  
Nov 30 15:04:57 ipa-2.mgmt.hss.int named-pkcs11[15661]: using 4 UDP listeners per interface                                                                                                   
Nov 30 15:04:57 ipa-2.mgmt.hss.int named-pkcs11[15661]: using up to 4096 sockets                                                                                                              
Nov 30 15:04:57 ipa-2.mgmt.hss.int named-pkcs11[15661]: ObjectStore.cpp(59): Failed to enumerate object store in /var/lib/softhsm/tokens/
Nov 30 15:04:57 ipa-2.mgmt.hss.int named-pkcs11[15661]: SoftHSM.cpp(456): Could not load the object store
Nov 30 15:04:57 ipa-2.mgmt.hss.int named-pkcs11[15661]: initializing DST: PKCS#11 initialization failed
Nov 30 15:04:57 ipa-2.mgmt.hss.int systemd[1]: named-pkcs11.service: control process exited, code=exited status=1
Nov 30 15:04:57 ipa-2.mgmt.hss.int systemd[1]: Failed to start Berkeley Internet Name Domain (DNS) with native PKCS#11.
Nov 30 15:04:57 ipa-2.mgmt.hss.int systemd[1]: Unit named-pkcs11.service entered failed state.
Nov 30 15:04:57 ipa-2.mgmt.hss.int systemd[1]: named-pkcs11.service failed.

Note You need to log in before you can comment on or make changes to this bug.