Bug 1300348 - crash of ipa-dnskeysync-replica and ipa-dnskeysncd
crash of ipa-dnskeysync-replica and ipa-dnskeysncd
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: IPA Maintainers
Namita Soman
Depends On:
  Show dependency treegraph
Reported: 2016-01-20 09:19 EST by Xiyang Dong
Modified: 2016-01-20 10:25 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1260663
Last Closed: 2016-01-20 10:25:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Xiyang Dong 2016-01-20 09:19:18 EST
+++ This bug was initially created as a clone of Bug #1260663 +++

Description of problem:
During automated execution of ipa-backup/restore feature, following two crashes seen.

:ipautil.py:373:run:CalledProcessError: Command ''/usr/libexec/ipa/ipa-dnskeysync-replica'' returned non-zero exit status 1
:Traceback (most recent call last):
:  File "/usr/libexec/ipa/ipa-dnskeysyncd", line 112, in <module>
:    while ldap_connection.syncrepl_poll(all=1, msgid=ldap_search):
:  File "/usr/lib64/python2.7/site-packages/ldap/syncrepl.py", line 405, in syncrepl_poll
:    self.syncrepl_refreshdone()
:  File "/usr/lib/python2.7/site-packages/ipapython/dnssec/keysyncer.py", line 113, in syncrepl_refreshdone
:    self.hsm_replica_sync()
:  File "/usr/lib/python2.7/site-packages/ipapython/dnssec/keysyncer.py", line 170, in hsm_replica_sync
:    ipautil.run([paths.IPA_DNSKEYSYNCD_REPLICA])
:  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 373, in run
:    raise CalledProcessError(p.returncode, arg_string, stdout)
:CalledProcessError: Command ''/usr/libexec/ipa/ipa-dnskeysync-replica'' returned non-zero exit status 1
:Local variables in innermost frame:
:p_in: None
:nolog: ()
:suplementary_groups: []
:preexec_fn: None
:arg_string: "'/usr/libexec/ipa/ipa-dnskeysync-replica'"
:stdout: ''
:p_out: -1
:p_err: -1
:runas: None
:stdin: None
:skip_output: False
:timeout: None
:capture_output: True
:p: <subprocess.Popen object at 0x6107f10>
:stderr: 'ipa: WARNING: session memcached servers not running\nipa         : DEBUG    Kerberos principal: ipa-dnskeysyncd/cloud-qe-3.testrelm.test\nipa         : DEBUG    Initializing principal ipa-dnskeysyncd/cloud-qe-3.testrelm.test using keytab /etc/ipa/dnssec/ipa-dnskeysyncd.keytab\nipa         : DEBUG    using ccache /tmp/ipa-dnskeysync-replica.ccache\nipa         : DEBUG    Attempt 1/5: success\nipa         : DEBUG    Got TGT\nipa         : DEBUG    Connecting to LDAP\nipa.ipaserver.plugins.ldap2.ldap2: DEBUG    Created connection context.ldap2_46913424\nipa         : DEBUG    Connected\nTraceback (most recent call last):\n  File "/usr/libexec/ipa/ipa-dnskeysync-replica", line 171, in <module>\n    open(paths.DNSSEC_SOFTHSM_PIN).read())\n  File "/usr/lib/python2.7/site-packages/ipapython/dnssec/localhsm.py", line 97, in __init__\n    self.p11 = _ipap11helper.P11_Helper(slot, pin, library)\n_ipap11helper.Error: Error at log in: 0xa0\n\nException AttributeError: "\'LocalHSM\' object has no attribute \'p11\'" in <bound method LocalHSM.__del__ of <ipapython.dnssec.localhsm.LocalHSM object at 0x47f1090>> ignored\n'
:raiseonerr: True
:env: {'LANG': 'en_US.UTF-8', 'SHELL': '/sbin/nologin', 'KRB5CCNAME': '/tmp/ipa-dnskeysyncd.ccache', 'LOGNAME': 'ods', 'USER': 'ods', 'SOFTHSM2_CONF': '/etc/ipa/dnssec/softhsm2.conf', 'PATH': '/bin:/sbin:/usr/kerberos/bin:/usr/kerberos/sbin:/usr/bin:/usr/sbin', 'HOME': '//var/lib/softhsm'}
:cwd: None
:args: ['/usr/libexec/ipa/ipa-dnskeysync-replica']

:localhsm.py:97:__init__:Error: Error at log in: 0xa0
:Traceback (most recent call last):
:  File "/usr/libexec/ipa/ipa-dnskeysync-replica", line 171, in <module>
:    open(paths.DNSSEC_SOFTHSM_PIN).read())
:  File "/usr/lib/python2.7/site-packages/ipapython/dnssec/localhsm.py", line 97, in __init__
:    self.p11 = _ipap11helper.P11_Helper(slot, pin, library)
:Error: Error at log in: 0xa0
:Local variables in innermost frame:
:slot: 0
:self: <ipapython.dnssec.localhsm.LocalHSM object at 0x47f1090>
:library: '/usr/lib64/pkcs11/libsofthsm2.so'
:pin: 'OGIfVEsRqtgbB6vQuWMzjcCcDedA1K'

Version-Release number of selected component (if applicable):
[root@dhcp207-229 ~]# rpm -q ipa-server
[root@dhcp207-229 ~]# 

How reproducible:

Steps to Reproduce:
1. Do ipa backup
2. ipa restore (full) from backup taken in step(1)

Actual results:
Crashes of ipa-dnskeysync-replica observed

Expected results:
No crash during ipa-restore process.

Additional info:

--- Additional comment from RHEL Product and Program Management on 2015-09-07 08:35:24 EDT ---

Since this bug report was entered in bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

--- Additional comment from Martin Bašti on 2015-09-09 11:20:44 EDT ---

Steps to reproduce:
1. server install
2. backup
3. server uninstall
4. server install
5. restore

Because server is installed, directory /var/lib/ipa/dnssec/tokens/ contains current tokens.

Restore adds there new tokens, but unfortunately old tokens are not removed, new tokens are just added into directory, and this cause issues with login.

--- Additional comment from Martin Bašti on 2015-09-10 07:40:02 EDT ---

Upstream ticket:

--- Additional comment from Martin Bašti on 2015-09-11 09:00:58 EDT ---

Fixed upstream

--- Additional comment from Petr Vobornik on 2015-09-11 09:21:48 EDT ---

Exception justification: needed for DNSSEC to work after restoration from a backup.

--- Additional comment from errata-xmlrpc on 2015-09-16 08:44:53 EDT ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2015:20912-01

--- Additional comment from Kaleem on 2015-10-01 10:08:53 EDT ---

Still observing the crash with ipa-4.2.0-12.el7

Please find the attached file with crash info.

--- Additional comment from Kaleem on 2015-10-01 10:09 EDT ---

--- Additional comment from Kaleem on 2015-10-16 02:40:10 EDT ---

Crash not seen with latest beaker runs of b&r feature, so turning it to verified state.

snip from beaker log:

 +-----------------------------[RPMs & OS: [RedHat - x86_64]-----------------------------+
|       ipa-admintools-4.2.0-15.el7.x86_64
|       ipa-client-4.2.0-15.el7.x86_64
|       ipa-server-4.2.0-15.el7.x86_64
|       ipa-server-dns-4.2.0-15.el7.x86_64
|       ipa-tests-ipa-server-rhel72-ipa-backup-restore-ksiddiqu-20150828120910-0.noarch
|       ipa-tests-ipa-server-rhel72-shared-20150930150523-0.noarch
|       sssd-ipa-1.13.0-40.el7.x86_64

     Test:[/ipa-server/rhel72/ipa-backup-restore/root]: [ Pass(8/8): 100% ] 
:: [   PASS   ]   ipa-backup_restore startup: Initial setup
:: [   PASS   ]   TC_001 :: IPA backup restore full
:: [   PASS   ]   TC_002 :: IPA backup restore full with gpg encryption/decryption related test cases
:: [   PASS   ]   TC_003 :: Data backup/restore related test cases
:: [   PASS   ]   TC_004 :: Data backup/restore backend/instance related test cases 
:: [   PASS   ]   TC_005 :: Additional test cases
:: [   PASS   ]   TC_006 :: Data restore from full backup related test cases
:: [   PASS   ]   /ipa-server/rhel72/ipa-backup-restore/root


--- Additional comment from errata-xmlrpc on 2015-11-19 07:06:19 EST ---

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

Comment 2 Xiyang Dong 2016-01-20 09:30 EST
Created attachment 1116664 [details]
crash info
Comment 3 Xiyang Dong 2016-01-20 09:32:27 EST
Saw similar issues on ipa-server-4.2.0-15 , attached full abrt emails above.
Comment 5 Martin Bašti 2016-01-20 09:45:31 EST
It looks like DS failed to start, or it is not ready yet.

NetworkError: cannot connect to 'ldapi://%2fvar%2frun%2fslapd-TESTRELM-TEST.socket':
Comment 6 Petr Spacek 2016-01-20 09:52:49 EST
The log description and attached log contain totally different traceback. In other words, this is NOT a duplicate/clone of Bug #1260663. In such cases please open a brand new bug and do not create clone, it just confuses things. Thank you!
Comment 7 Xiyang Dong 2016-01-20 10:25:40 EST
Yeah I agree, Opened https://bugzilla.redhat.com/show_bug.cgi?id=1300372 and mark this one closed.Thanks.

Note You need to log in before you can comment on or make changes to this bug.