Description of problem:
Multiple processes concurrently invoking krb5_cc_initialize() with the same ccache filename will sometimes fail with KRB5_FCC_INTERNAL.

Version-Release number of selected component (if applicable):
krb5-libs-1.14.1-1.fc25.x86_64
but this was originally observed on RHEL6 and it looks like it affects all krb5 releases.

How reproducible:
quite easily with a synthetic reproducer

Steps to Reproduce:
cat >krb5initrace.c <<EOF
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <krb5.h>

int main(int argc, char *argv[]) {
    krb5_context ctx;
    krb5_init_context(&ctx);
    krb5_ccache ccache;
    krb5_cc_default(ctx, &ccache);
    krb5_principal princ;
    krb5_parse_name(ctx, "example", &princ);

    unsigned int attempt = 0;
    while (1) {
        attempt ++;
        int result = krb5_cc_initialize(ctx, ccache, princ);
        if (result != 0) {
            fprintf(stderr, "pid %d attempt %u failed: %d\n", getpid(), attempt, result);
            return 1;
        }
    }
}
EOF
gcc -o krb5initrace $(pkg-config krb5 --libs) krb5initrace.c
export KRB5CCNAME=/tmp/asdf
./krb5initrace & ./krb5initrace &

Actual results:
One of the two processes will quickly fail with KRB5_FCC_INTERNAL (-1765328188), as in:
pid 2415 attempt 6 failed: -1765328188

Expected results:
krb5_cc_initialize() should be safe to call on the same ccache filename from multiple processes concurrently without random crashes.

Additional info:
The API docs don't make any explicit mention about file-backed ccache safety when shared across multiple processes, but from poking around in the source I can see that cc_file.c does go to a lot of effort to ensure that a suitable POSIX file lock is held on the ccache filename whenever it's manipulated. However the ccache initialization routine is a bit of a special case in that it tries to first erase and then recreate the ccache filename. It does that by calling unlink() followed by open() with O_CREAT|O_EXCL, and then it acquires a lock and writes to the file. The problem is there is a race window between unlink() and open() where a racing process may already have created the same file, in which case open() will fail with EEXIST. That gets mapped to KRB5_FCC_INTERNAL which is what we are seeing.

We originally saw this in the bkr command line client, which can be configured to initialize the ccache using a keytab before authenticating. Some users reported hitting this -1765328188 error at random when they had multiple Jenkins jobs running concurrently under the same user account sharing the same default ccache file on RHEL6. That's bug 1313580.

The race between unlink() and open() seems to exist in all versions of krb5, including back to RHEL6 where we originally hit this, but I've filed this bug against Fedora rawhide because the krb5 fix for this would probably be a bit too drastic to be backported to RHEL6.