Bug 1662222 - [abrt] bind-pkcs11: k5_mutex_lock(): named-pkcs11 killed by SIGABRT
Summary: [abrt] bind-pkcs11: k5_mutex_lock(): named-pkcs11 killed by SIGABRT
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: krb5
Version: rawhide
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Robbie Harwood
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:821895e572846cbffa98c63d917...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-27 06:33 UTC by Brandon Johnson
Modified: 2019-12-03 19:27 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-27 17:45:25 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: backtrace (18.61 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: cgroup (186 bytes, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: core_backtrace (11.75 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: cpuinfo (1.10 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: dso_list (5.59 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: environ (242 bytes, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: limits (1.29 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: maps (33.55 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: mountinfo (3.89 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: open_fds (1.25 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: proc_pid_status (1.28 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details
File: var_log_messages (7.04 KB, text/plain)
2018-12-27 06:33 UTC, Brandon Johnson
no flags Details

Description Brandon Johnson 2018-12-27 06:33:09 UTC
Description of problem:
this occurred shortly after an update to some 389 and krb packages on my FreeIPA instance

Version-Release number of selected component:
32:bind-pkcs11-9.11.4-10.P2.fc29

Additional info:
reporter:       libreport-2.9.7
backtrace_rating: 3
cmdline:        /usr/sbin/named-pkcs11 -u named -c /etc/named.conf
crash_function: k5_mutex_lock
executable:     /usr/sbin/named-pkcs11
journald_cursor: s=877760977bfa4ff7b3b7015e240bccb6;i=135a06;b=054b6d092f934ec7b75c91cc550dc679;m=10031db;t=57df16ba83402;x=319798af0667996b
kernel:         4.19.10-300.fc29.x86_64
rootdir:        /
runlevel:       unknown
type:           CCpp
uid:            25

Comment 1 Brandon Johnson 2018-12-27 06:33:12 UTC
Created attachment 1516988 [details]
File: backtrace

Comment 2 Brandon Johnson 2018-12-27 06:33:13 UTC
Created attachment 1516989 [details]
File: cgroup

Comment 3 Brandon Johnson 2018-12-27 06:33:14 UTC
Created attachment 1516990 [details]
File: core_backtrace

Comment 4 Brandon Johnson 2018-12-27 06:33:16 UTC
Created attachment 1516991 [details]
File: cpuinfo

Comment 5 Brandon Johnson 2018-12-27 06:33:17 UTC
Created attachment 1516992 [details]
File: dso_list

Comment 6 Brandon Johnson 2018-12-27 06:33:19 UTC
Created attachment 1516993 [details]
File: environ

Comment 7 Brandon Johnson 2018-12-27 06:33:20 UTC
Created attachment 1516994 [details]
File: limits

Comment 8 Brandon Johnson 2018-12-27 06:33:21 UTC
Created attachment 1516995 [details]
File: maps

Comment 9 Brandon Johnson 2018-12-27 06:33:22 UTC
Created attachment 1516996 [details]
File: mountinfo

Comment 10 Brandon Johnson 2018-12-27 06:33:22 UTC
Created attachment 1516997 [details]
File: open_fds

Comment 11 Brandon Johnson 2018-12-27 06:33:23 UTC
Created attachment 1516998 [details]
File: proc_pid_status

Comment 12 Brandon Johnson 2018-12-27 06:33:24 UTC
Created attachment 1516999 [details]
File: var_log_messages

Comment 13 Ben Cotton 2019-10-31 20:15:32 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 14 Petr Menšík 2019-11-27 12:45:12 UTC
Oh, this issue is quite strange. Deep inside ldap library it crashes on kerberos assertion. I cannot say I understand how it hit such state. But anyway, I think this might be more related to bind-dyndb-ldap issue.

Comment 15 Petr Menšík 2019-11-27 12:51:02 UTC
Do you think you can identify this issue and whether it is already fixed? It happened just long time ago.

Comment 16 Alexander Bokovoy 2019-11-27 13:24:49 UTC
It fails inside krb5 memory ccache routine. Robbie, if you have any idea how to address that in bind-dyndb-ldap, I'm all ears.

bind-dyndb-ldap uses LDAP bind with Cyrus-SASL via ldap_sasl_interactive_bind_s() this way:

        case AUTH_SASL:
                CHECK(setting_get_str("sasl_mech", ldap_inst->local_settings,
                                      &sasl_mech));
                if (strcmp(sasl_mech, "GSSAPI") == 0) {
                        CHECK(setting_get_str("krb5_principal",
                                              ldap_inst->local_settings,
                                              &krb5_principal));
                        CHECK(setting_get_str("krb5_keytab",
                                              ldap_inst->local_settings,
                                              &krb5_keytab));
                        LOCK(&ldap_inst->kinit_lock);
                        result = get_krb5_tgt(ldap_inst->mctx,
                                              krb5_principal,
                                              krb5_keytab);
                        UNLOCK(&ldap_inst->kinit_lock);
                        if (result != ISC_R_SUCCESS)
                                return ISC_R_NOTCONNECTED;
                }

                log_debug(4, "trying interactive bind using '%s' mechanism",
                          sasl_mech);
                ret = ldap_sasl_interactive_bind_s(ldap_conn->handle, NULL,
                                                   sasl_mech,
                                                   NULL, NULL, LDAP_SASL_QUIET,
                                                   ldap_sasl_interact,
                                                   ldap_inst);
                break;

where get_krb5_tgt() creates in-memory CCACHE and sets its name in KRB5CCNAME environmental variable. One strange thing is that it does
        /* get credentials cache */
        CHECK(str_new(mctx, &ccname));
        CHECK(str_sprintf(ccname, "MEMORY:_ld_krb5_cc_%s", principal));
        ret = setenv("KRB5CCNAME", str_buf(ccname), 1);

        krberr = krb5_cc_resolve(context, str_buf(ccname), &ccache);
        krberr = krb5_kt_resolve(context, keyfile, &keytab);
        krberr = krb5_get_init_creds_keytab(context, &my_creds, kprincpw,
                                            keytab, 0, NULL, &options);
        krberr = krb5_cc_initialize(context, ccache, kprincpw);
        krberr = krb5_cc_store_cred(context, ccache, &my_creds);

and then upon completion it clears them all:
        if (ccache) krb5_cc_close(context, ccache);
        if (keytab) krb5_kt_close(context, keytab);
        if (kprincpw) krb5_free_principal(context, kprincpw);
        if (my_creds_ptr) krb5_free_cred_contents(context, my_creds_ptr);
        if (context) krb5_free_context(context);

So even with successful acquisition the ccache will be destroyed and for in-memory ccache we'll get the ccache destroyed, its content emptied if no other references exist to it. I can definitely see cases when there are no other threads referencing the same ccache anymore.

Comment 17 Ben Cotton 2019-11-27 14:16:12 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 18 Ben Cotton 2019-11-27 14:35:06 UTC
This bug was accidentally closed due to a query error. Reopening.

Comment 19 Robbie Harwood 2019-11-27 17:45:25 UTC
This looks very similar to https://bugzilla.redhat.com/show_bug.cgi?id=1633089 and a few others like it.  I think these were due to a bug in our handling of memory ccache access by multiple handles concurrently - this has since been fixed, so I'm going to close this out (fixed by 146dadec8fe7ccc4149eb2e3f577cc320aee6efb and then later by moving to a new design in 088ba228acce4fd55bbb7c30122fe2703b8beeb8 )

In general, traces that hit an assertion below k5_mutex_lock() are indicative of corruption of some kind.  The lock structure is clobbered with garbage (in this case, by being reallocated), and glibc's sentinel is overwritten.


Note You need to log in before you can comment on or make changes to this bug.