Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1401341 - Using curl with SPNEG0 Authentication Creates Assertion Failure
Using curl with SPNEG0 Authentication Creates Assertion Failure
Status: CLOSED DUPLICATE of bug 1398370
Product: Fedora
Classification: Fedora
Component: gssntlmssp (Show other bugs)
25
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Simo Sorce
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-12-04 18:20 EST by Ryan Ware
Modified: 2016-12-05 15:50 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-05 15:47:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Verbose curl output (6.96 KB, text/plain)
2016-12-05 13:28 EST, Ryan Ware
no flags Details
strace output (1.02 MB, text/plain)
2016-12-05 13:28 EST, Ryan Ware
no flags Details
ltrace output (469.03 KB, text/plain)
2016-12-05 13:29 EST, Ryan Ware
no flags Details

  None (edit)
Description Ryan Ware 2016-12-04 18:20:47 EST
Description of problem: Using curl with --negotiate for SPNEG0 authentication causes curl to return error 127 and the message: "Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion `map->l_init_called' failed!"


Version-Release number of selected component (if applicable): curl 7.51.0 (x86_64-redhat-linux-gnu) libcurl/7.51.0 NSS/3.27 zlib/1.2.8 libidn2/0.11 libpsl/0.14.0 (+libidn2/0.10) libssh2/1.8.0 nghttp2/1.13.0


How reproducible: 100%


Steps to Reproduce:
1. curl --negotiate -u : https://url-with-krb-authentication

Actual results:
Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion `map->l_init_called' failed!


Expected results:
Content of web site.

Additional info:

I'm guessing this is really an issue with the crypto libraries that curl links to and not curl itself.  The error is an odd one.  Googling seems to indicate that it's related to not handling plugin deallocation correction.  I have no idea if this is the case.  That said, looking at the stack trace, right before the writev() spitting out the "Inconsistency detected...", there is a munmap() corresponding to the /lib64/libnsssysinit.so library.  Shortly before that, there is also a munmap() corresponding to the /lib64/libnsspem.so library.
Comment 1 Kamil Dudka 2016-12-05 03:24:01 EST
I am not able to reproduce it, works just fine on my f25 box:

$ rpm -q glibc curl libcurl nss nss-softokn
glibc-2.24-3.fc25.x86_64
glibc-2.24-3.fc25.i686
curl-7.51.0-3.fc25.x86_64
libcurl-7.51.0-3.fc25.x86_64
nss-3.27.0-1.2.fc25.x86_64
nss-softokn-3.27.0-1.0.fc25.x86_64

$ curl -svo/dev/null https://url-with-krb-authentication/object --negotiate -L -u :
*   Trying 10.19.208.25...
* TCP_NODELAY set
* Connected to url-with-krb-authentication (10.19.208.25) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: [...]
> GET /object HTTP/1.1
> Host: url-with-krb-authentication
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Date: Mon, 05 Dec 2016 08:18:06 GMT
< Server: Apache/2.4.18 (Red Hat) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< WWW-Authenticate: Negotiate
< WWW-Authenticate: Basic realm="Kerberos 5 Login"
< Content-Length: 381
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
{ [381 bytes data]
* Curl_http_done: called premature == 0
* Connection #0 to host url-with-krb-authentication left intact
* Issue another request to this URL: 'https://url-with-krb-authentication/object'
* Found bundle for host url-with-krb-authentication: 0x555c9b344160 [can pipeline]
* Re-using existing connection! (#0) with host url-with-krb-authentication
* Connected to url-with-krb-authentication (10.19.208.25) port 443 (#0)
* Server auth using Negotiate with user ''
> GET /object HTTP/1.1
> Host: url-with-krb-authentication
> Authorization: Negotiate [...]
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 05 Dec 2016 08:18:07 GMT
< Server: Apache/2.4.18 (Red Hat) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50
< WWW-Authenticate: Negotiate [...]
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< X-UA-Compatible: IE=Edge,chrome=1
< Cache-Control: must-revalidate, private, max-age=0
< X-Request-Id: f541b1201b6773ebdbe3cb6ebf20d78a
< X-Runtime: 0.796864
< X-Rack-Cache: miss
< X-Powered-By: Phusion Passenger 4.0.50
< ETag: "397879774e4ca607a2eaad2e6a206085"
< Status: 200 OK
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< 
{ [6 bytes data]
* Curl_http_done: called premature == 0
* Closing connection 0
Comment 2 David Woodhouse 2016-12-05 04:43:51 EST
Ryan, please can you show that again with '-v' and also show the full stack trace? Without those I can't even see how it's authenticating — is it using Kerberos or NTLM, etc.?

Repeating it in valgrind might help too.
Comment 3 Ryan Ware 2016-12-05 13:28 EST
Created attachment 1228127 [details]
Verbose curl output
Comment 4 Ryan Ware 2016-12-05 13:28 EST
Created attachment 1228128 [details]
strace output
Comment 5 Ryan Ware 2016-12-05 13:29 EST
Created attachment 1228129 [details]
ltrace output
Comment 6 Ryan Ware 2016-12-05 13:33:03 EST
I was hoping to avoid posting stack traces and verbose output because I want to be as careful as possible not to accidentally leak Intel infrastructure information.  That said, I've attached verbose output from curl and stack traces (both ltrace and strace for those who have a preference).  None of the three files is *exactly* what the tools spit out.  Each has Intel infrastructure or other sensitive information redacted but the changes shouldn't preclude debugging.
Comment 7 David Woodhouse 2016-12-05 14:52:32 EST
Please don't over-sanitise. There's no harm in people knowing that you are talking to pki.intel.com.

The last munmap from your ltrace appears to be NSS's libnsssysinit.so:

open("/lib64/libnsssysinit.so", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\f\0\0\0\0\0\0"..., 832) = 832
fstat(4, {st_mode=S_IFREG|0755, st_size=11360, ...}) = 0
mmap(NULL, 2105544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7fa97f2cc000

...

munmap(0x7fa97f2cc000, 2105544)         = 0
writev(2, [{iov_base="Inconsistency detected by ld.so:"..., iov_len=33}, {iov_base="dl-close.c", iov_len=10}, {iov_base=": ", iov_len=2}, {iov_base="811", iov_len=3}, {iov_base=": ", iov_len=2}, {iov_base="_dl_close", iov_len=9}, {iov_base=": ", iov_len=2}, {iov_base="Assertion `", iov_len=11}, {iov_base="map->l_init_called", iov_len=18}, {iov_base="' failed!\n", iov_len=10}], 10) = 100

I don't see the requested backtrace, and I'm not sure if we *successfully* unmapped libnsssysinit.so and then hit that assertion when attempting to dlclose() the *next* library. But can you try running 'sudo setup-nsssysinit.sh off' and see if the problem goes away?
Comment 8 David Woodhouse 2016-12-05 15:07:40 EST
You weren't using NTLMSSP; your log shows a successful Kerberos authentication, not a fallback to NTLM (thanks for not over-sanitising the WWW-Authenticate: lines despite Kamil's dubious precedent above). But just in case, can you also try moving /etc/gss/mech.d/ntlmssp.conf out of the way and see if it still happens then?
Comment 9 Ryan Ware 2016-12-05 15:28:41 EST
(In reply to David Woodhouse from comment #7)
> Please don't over-sanitise. There's no harm in people knowing that you are
> talking to pki.intel.com.

If I hadn't *explicitly* been told by InfoSec to not reveal the urls, I would have left it in.

> The last munmap from your ltrace appears to be NSS's libnsssysinit.so:
> 
> open("/lib64/libnsssysinit.so", O_RDONLY|O_CLOEXEC) = 4
> read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\f\0\0\0\0\0\0"...,
> 832) = 832
> fstat(4, {st_mode=S_IFREG|0755, st_size=11360, ...}) = 0
> mmap(NULL, 2105544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) =
> 0x7fa97f2cc000
> 
> ...
> 
> munmap(0x7fa97f2cc000, 2105544)         = 0
> writev(2, [{iov_base="Inconsistency detected by ld.so:"..., iov_len=33},
> {iov_base="dl-close.c", iov_len=10}, {iov_base=": ", iov_len=2},
> {iov_base="811", iov_len=3}, {iov_base=": ", iov_len=2},
> {iov_base="_dl_close", iov_len=9}, {iov_base=": ", iov_len=2},
> {iov_base="Assertion `", iov_len=11}, {iov_base="map->l_init_called",
> iov_len=18}, {iov_base="' failed!\n", iov_len=10}], 10) = 100
> 
> I don't see the requested backtrace, and I'm not sure if we *successfully*
> unmapped libnsssysinit.so and then hit that assertion when attempting to
> dlclose() the *next* library. But can you try running 'sudo
> setup-nsssysinit.sh off' and see if the problem goes away?

This does not fix the problem.
Comment 10 Ryan Ware 2016-12-05 15:29:56 EST
(In reply to David Woodhouse from comment #8)
> You weren't using NTLMSSP; your log shows a successful Kerberos
> authentication, not a fallback to NTLM (thanks for not over-sanitising the
> WWW-Authenticate: lines despite Kamil's dubious precedent above). But just
> in case, can you also try moving /etc/gss/mech.d/ntlmssp.conf out of the way
> and see if it still happens then?

It no longer happens after removing /etc/gss/mech.d/ntlmssp.conf.

It's default, but my ntlmssp.conf contains the following:

# NTLMSSP mechanism plugin
#
# NOTE: to activate the NTLM SSP mechanism do the following
#     * FOR krb5 < 1.12: copy the content of this file in /etc/gss/mech
#     * FOR krb5 >= 1.12: copy this file in /etc/gss/mech.d
#
# Mechanism Name        Object Identifier               Shared Library Path                     Other Options
gssntlmssp_v1           1.3.6.1.4.1.311.2.2.10          /usr/lib64/gssntlmssp/gssntlmssp.so
Comment 11 David Woodhouse 2016-12-05 15:42:31 EST
(In reply to Ryan Ware from comment #9) 
> If I hadn't *explicitly* been told by InfoSec to not reveal the urls, I
> would have left it in.

They are very silly. That URL is entirely harmless, and *also* present in this bugzilla and others many times already — they never told *me* that :)

(In reply to Ryan Ware from comment #10)
> It no longer happens after removing /etc/gss/mech.d/ntlmssp.conf.

OK, let's blame Simo...
Comment 12 Simo Sorce 2016-12-05 15:47:32 EST
Sounds like another duplicate of 1398370

*** This bug has been marked as a duplicate of bug 1398370 ***
Comment 13 Ryan Ware 2016-12-05 15:50:55 EST
(In reply to Simo Sorce from comment #12)
> Sounds like another duplicate of 1398370

Agreed.  Thought I searched appropriately.

Note You need to log in before you can comment on or make changes to this bug.