Description of problem: Using curl with --negotiate for SPNEG0 authentication causes curl to return error 127 and the message: "Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion `map->l_init_called' failed!" Version-Release number of selected component (if applicable): curl 7.51.0 (x86_64-redhat-linux-gnu) libcurl/7.51.0 NSS/3.27 zlib/1.2.8 libidn2/0.11 libpsl/0.14.0 (+libidn2/0.10) libssh2/1.8.0 nghttp2/1.13.0 How reproducible: 100% Steps to Reproduce: 1. curl --negotiate -u : https://url-with-krb-authentication Actual results: Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion `map->l_init_called' failed! Expected results: Content of web site. Additional info: I'm guessing this is really an issue with the crypto libraries that curl links to and not curl itself. The error is an odd one. Googling seems to indicate that it's related to not handling plugin deallocation correction. I have no idea if this is the case. That said, looking at the stack trace, right before the writev() spitting out the "Inconsistency detected...", there is a munmap() corresponding to the /lib64/libnsssysinit.so library. Shortly before that, there is also a munmap() corresponding to the /lib64/libnsspem.so library.
I am not able to reproduce it, works just fine on my f25 box: $ rpm -q glibc curl libcurl nss nss-softokn glibc-2.24-3.fc25.x86_64 glibc-2.24-3.fc25.i686 curl-7.51.0-3.fc25.x86_64 libcurl-7.51.0-3.fc25.x86_64 nss-3.27.0-1.2.fc25.x86_64 nss-softokn-3.27.0-1.0.fc25.x86_64 $ curl -svo/dev/null https://url-with-krb-authentication/object --negotiate -L -u : * Trying 10.19.208.25... * TCP_NODELAY set * Connected to url-with-krb-authentication (10.19.208.25) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * ALPN/NPN, server did not agree to a protocol * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: [...] > GET /object HTTP/1.1 > Host: url-with-krb-authentication > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 401 Unauthorized < Date: Mon, 05 Dec 2016 08:18:06 GMT < Server: Apache/2.4.18 (Red Hat) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50 < Strict-Transport-Security: max-age=63072000; includeSubDomains < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < WWW-Authenticate: Negotiate < WWW-Authenticate: Basic realm="Kerberos 5 Login" < Content-Length: 381 < Content-Type: text/html; charset=iso-8859-1 < * Ignoring the response-body { [381 bytes data] * Curl_http_done: called premature == 0 * Connection #0 to host url-with-krb-authentication left intact * Issue another request to this URL: 'https://url-with-krb-authentication/object' * Found bundle for host url-with-krb-authentication: 0x555c9b344160 [can pipeline] * Re-using existing connection! (#0) with host url-with-krb-authentication * Connected to url-with-krb-authentication (10.19.208.25) port 443 (#0) * Server auth using Negotiate with user '' > GET /object HTTP/1.1 > Host: url-with-krb-authentication > Authorization: Negotiate [...] > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 200 OK < Date: Mon, 05 Dec 2016 08:18:07 GMT < Server: Apache/2.4.18 (Red Hat) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50 < WWW-Authenticate: Negotiate [...] < X-Content-Type-Options: nosniff < X-Frame-Options: SAMEORIGIN < X-XSS-Protection: 1; mode=block < Strict-Transport-Security: max-age=63072000; includeSubDomains < X-UA-Compatible: IE=Edge,chrome=1 < Cache-Control: must-revalidate, private, max-age=0 < X-Request-Id: f541b1201b6773ebdbe3cb6ebf20d78a < X-Runtime: 0.796864 < X-Rack-Cache: miss < X-Powered-By: Phusion Passenger 4.0.50 < ETag: "397879774e4ca607a2eaad2e6a206085" < Status: 200 OK < Vary: Accept-Encoding < Transfer-Encoding: chunked < Content-Type: text/html; charset=utf-8 < { [6 bytes data] * Curl_http_done: called premature == 0 * Closing connection 0
Ryan, please can you show that again with '-v' and also show the full stack trace? Without those I can't even see how it's authenticating — is it using Kerberos or NTLM, etc.? Repeating it in valgrind might help too.
Created attachment 1228127 [details] Verbose curl output
Created attachment 1228128 [details] strace output
Created attachment 1228129 [details] ltrace output
I was hoping to avoid posting stack traces and verbose output because I want to be as careful as possible not to accidentally leak Intel infrastructure information. That said, I've attached verbose output from curl and stack traces (both ltrace and strace for those who have a preference). None of the three files is *exactly* what the tools spit out. Each has Intel infrastructure or other sensitive information redacted but the changes shouldn't preclude debugging.
Please don't over-sanitise. There's no harm in people knowing that you are talking to pki.intel.com. The last munmap from your ltrace appears to be NSS's libnsssysinit.so: open("/lib64/libnsssysinit.so", O_RDONLY|O_CLOEXEC) = 4 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\f\0\0\0\0\0\0"..., 832) = 832 fstat(4, {st_mode=S_IFREG|0755, st_size=11360, ...}) = 0 mmap(NULL, 2105544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7fa97f2cc000 ... munmap(0x7fa97f2cc000, 2105544) = 0 writev(2, [{iov_base="Inconsistency detected by ld.so:"..., iov_len=33}, {iov_base="dl-close.c", iov_len=10}, {iov_base=": ", iov_len=2}, {iov_base="811", iov_len=3}, {iov_base=": ", iov_len=2}, {iov_base="_dl_close", iov_len=9}, {iov_base=": ", iov_len=2}, {iov_base="Assertion `", iov_len=11}, {iov_base="map->l_init_called", iov_len=18}, {iov_base="' failed!\n", iov_len=10}], 10) = 100 I don't see the requested backtrace, and I'm not sure if we *successfully* unmapped libnsssysinit.so and then hit that assertion when attempting to dlclose() the *next* library. But can you try running 'sudo setup-nsssysinit.sh off' and see if the problem goes away?
You weren't using NTLMSSP; your log shows a successful Kerberos authentication, not a fallback to NTLM (thanks for not over-sanitising the WWW-Authenticate: lines despite Kamil's dubious precedent above). But just in case, can you also try moving /etc/gss/mech.d/ntlmssp.conf out of the way and see if it still happens then?
(In reply to David Woodhouse from comment #7) > Please don't over-sanitise. There's no harm in people knowing that you are > talking to pki.intel.com. If I hadn't *explicitly* been told by InfoSec to not reveal the urls, I would have left it in. > The last munmap from your ltrace appears to be NSS's libnsssysinit.so: > > open("/lib64/libnsssysinit.so", O_RDONLY|O_CLOEXEC) = 4 > read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\f\0\0\0\0\0\0"..., > 832) = 832 > fstat(4, {st_mode=S_IFREG|0755, st_size=11360, ...}) = 0 > mmap(NULL, 2105544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = > 0x7fa97f2cc000 > > ... > > munmap(0x7fa97f2cc000, 2105544) = 0 > writev(2, [{iov_base="Inconsistency detected by ld.so:"..., iov_len=33}, > {iov_base="dl-close.c", iov_len=10}, {iov_base=": ", iov_len=2}, > {iov_base="811", iov_len=3}, {iov_base=": ", iov_len=2}, > {iov_base="_dl_close", iov_len=9}, {iov_base=": ", iov_len=2}, > {iov_base="Assertion `", iov_len=11}, {iov_base="map->l_init_called", > iov_len=18}, {iov_base="' failed!\n", iov_len=10}], 10) = 100 > > I don't see the requested backtrace, and I'm not sure if we *successfully* > unmapped libnsssysinit.so and then hit that assertion when attempting to > dlclose() the *next* library. But can you try running 'sudo > setup-nsssysinit.sh off' and see if the problem goes away? This does not fix the problem.
(In reply to David Woodhouse from comment #8) > You weren't using NTLMSSP; your log shows a successful Kerberos > authentication, not a fallback to NTLM (thanks for not over-sanitising the > WWW-Authenticate: lines despite Kamil's dubious precedent above). But just > in case, can you also try moving /etc/gss/mech.d/ntlmssp.conf out of the way > and see if it still happens then? It no longer happens after removing /etc/gss/mech.d/ntlmssp.conf. It's default, but my ntlmssp.conf contains the following: # NTLMSSP mechanism plugin # # NOTE: to activate the NTLM SSP mechanism do the following # * FOR krb5 < 1.12: copy the content of this file in /etc/gss/mech # * FOR krb5 >= 1.12: copy this file in /etc/gss/mech.d # # Mechanism Name Object Identifier Shared Library Path Other Options gssntlmssp_v1 1.3.6.1.4.1.311.2.2.10 /usr/lib64/gssntlmssp/gssntlmssp.so
(In reply to Ryan Ware from comment #9) > If I hadn't *explicitly* been told by InfoSec to not reveal the urls, I > would have left it in. They are very silly. That URL is entirely harmless, and *also* present in this bugzilla and others many times already — they never told *me* that :) (In reply to Ryan Ware from comment #10) > It no longer happens after removing /etc/gss/mech.d/ntlmssp.conf. OK, let's blame Simo...
Sounds like another duplicate of 1398370 *** This bug has been marked as a duplicate of bug 1398370 ***
(In reply to Simo Sorce from comment #12) > Sounds like another duplicate of 1398370 Agreed. Thought I searched appropriately.