Bug 1366801

copy/paste the wrong curl command: --ntlm should be --proxy-basic:

  $ curl --proxy-basic -L --connect-timeout 5 -x ...

Please can you test it with dnf 2.0 - meny things were changed including conf parser? The version is placed in copr repository. Tu enable copr repository run ``dnf copr enable rpmsoftwaremanagement/dnf-nightly``

Then you will be able to upgrade to dnf 2.0.

Please can you report if the problem was solved by the new version?

Thanks a lot.

Updating dnf from copr repo rpmsoftwaremanagement/dnf-nightly did not fix the issue:

$ rpm -q dnf
dnf-2.0.0-0.288g1e2dc8a.fc24.noarch


With the same settings from comment #0

$ sudo dnf -vv update
Loaded plugins: builddep, config-manager, copr, debuginfo-install, download, generate_completion_cache, needs-restarting, noroot, playground, reposync
DNF version: 2.0.0
cachedir: /var/cache/dnf
Unknown configuration option: enabled_metadata = 1
Unknown configuration option: enabled_metadata = 1
Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f24&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f24&arch=x86_64 [Received HTTP code 407 from proxy after CONNECT].
Error: Failed to synchronize cache for repo 'updates'

In dnf/repo.py if LRO_PROXYAUTH is set to True:

  ...
  h.setopt(librepo.LRO_PROXYAUTH, True)
  ...

Would this cause librepo to not use BASIC auth when dealing with a proxy?

In librepo's librepo/handle.c:

    ....
    case LRO_PROXYAUTH:                                                         
        if (va_arg(arg, long) == 0) {                                           
            c_rc = curl_easy_setopt(c_h, CURLOPT_PROXYAUTH, CURLAUTH_BASIC);    
            handle->proxyauthmethods = LR_AUTH_BASIC;                           
        } else {                                                                
            c_rc = curl_easy_setopt(c_h, CURLOPT_PROXYAUTH, CURLAUTH_ANY);      
            handle->proxyauthmethods = LR_AUTH_ANY;                             
        }                                                                       
        break;
    ....

Does this mean the curl's anyauth is being set?  Testing curl with --proxy-anyauth fails since it defaults to NTLM.

With Comment #0 curl command using --proxy-anyauth it appears it use NTLM with GSS:
  
  ...
  
  ...
  < HTTP/1.1 407 Proxy Authorization Required
  < Date: Mon, 22 Aug 2016 17:48:54 GMT
  < Proxy-Connection: keep-alive
  < Via: 1.1 localhost.localdomain
  < Cache-Control: no-store
  < Content-Type: text/html
  < Content-Language: en
  < Proxy-Authenticate: Negotiate
  * gss_init_sec_context() failed: : SPNEGO cannot find mechanisms to negotiate
  < Proxy-Authenticate: NTLM
  < Content-Length: 666
  < 
  * Received HTTP code 407 from proxy after CONNECT
    0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
  * Closing connection 0
  curl: (56) Received HTTP code 407 from proxy after CONNECT




Is this related?  https://github.com/curl/curl/issues/876

It looks like a librepo/curl issue -> reassigning.

Maybe, but shouldn't dnf be using LRO_PROXYAUTHMETHODS base off https://github.com/rpm-software-management/librepo/commit/bfc05df2f74ad7038977ce4c6b1fd367ec1da430#diff-9cff5beb4a457ecb5636a70db528d172R157

In that case you can choose which auth types the proxy should be using or at least let the user override what librepo/curl wants to do?

I'm on Fed 25 and experiencing exactly this same behavior.

In addition i want to note that it _seems_ that dnf is not picking up the CA certificates, where as curl does:
dnf -V
timer: config: 6 ms
cachedir: /var/cache/dnf
Loaded plugins: builddep, protected_packages, debuginfo-install, needs-restarting, playground, generate_completion_cache, noroot, config-manager, download, copr, reposync, Query
DNF version: 1.1.10
Command: dnf -V 
Installroot: /
Releasever: 25

cat /etc/dnf/dnf.conf 
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True

proxy=http://<IP>:8002
proxy_username=<username>
proxy_password=<pass>
#sslverify=0
sslverify=True
sslcacert=/etc/pki/ca-trust/source/
cacert=/etc/pki/ca-trust/source/
sslclientcert=/etc/pki/ca-trust/source/
debuglevel=10


dnf -vv install vboot-utils

from dnf.librepo:

12:39:31 Librepo version: 1.7.18 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.51.0 NSS/3.29.3 zlib/1.2.8 libidn2/2.0.0 libpsl/0.17.0 (+libidn2/0.11) libssh2/1.8.0 nghttp2/1.13.0)
12:39:31 Current date: 2017-04-27T12:39:31-0400
12:39:31 lr_download: Target: file:///etc/dnf/dnf.conf (-)
12:39:31 select_next_target: Selecting mirror for: file:///etc/dnf/dnf.conf
12:39:31 prepare_next_transfer: URL: file:///etc/dnf/dnf.conf
12:39:31 add_librepo_xattr: Cannot set xattr user.Librepo.DownloadInProgress (fd: 7): Operation not supported

<skip>
12:39:32 prepare_next_transfer: URL: https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64
12:39:32 add_librepo_xattr: Cannot set xattr user.Librepo.DownloadInProgress (fd: 10): Operation not supported
12:39:32 lr_download: Downloading started
12:39:32 check_transfer_statuses: Transfer finished: https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 (Effective url: https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64)
12:39:32 check_transfer_statuses: Error during transfer: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 [Received HTTP code 407 from proxy after CONNECT]
12:39:32 check_transfer_statuses: No more retries (tried: 1)
12:39:32 lr_download: Error while downloading: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 [Received HTTP code 407 from proxy after CONNECT]
12:39:32 lr_handle_prepare_internal_mirrorlist: LRO_METALINKURL processing failed
12:39:32 Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 [Received HTTP code 407 from proxy after CONNECT]


If i enable the following params in the dnf.conf:
#sslverify=0
sslverify=True
sslcacert=/etc/pki/ca-trust/source/
cacert=/etc/pki/ca-trust/source/
sslclientcert=/etc/pki/ca-trust/source/

i'll get :
<skip>
14:11:01 add_librepo_xattr: Cannot set xattr user.Librepo.DownloadInProgress (fd: 12): Operation not supported
14:11:01 lr_download: Downloading started
14:11:01 check_transfer_statuses: Transfer finished: https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 (Effective url: https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64)
14:11:01 check_finished_transfer_status: Fatal error - Curl code (77): Problem with the SSL CA cert (path? access rights?) for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 []
14:11:01 check_transfer_statuses: Error during transfer: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 []
14:11:01 check_transfer_statuses: No more retries (tried: 1)
14:11:01 lr_download: Error while downloading: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 []
14:11:01 lr_handle_prepare_internal_mirrorlist: LRO_METALINKURL processing failed
14:11:01 Cannot prepare internal mirrorlist: Curl error (77): Problem with the SSL CA cert (path? access rights?) for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f25&arch=x86_64 []



so i'd say that there are 2 issues:
 a. DNF does NOT pick up NTLM authentication, though curl works fine.
 b. DNF does NOT pick the location of the CA certificates, though curl does.

to test with curl :
<as root from /root>
cat ./.curlrc 
proxy=http://<user>:<pass>@<IP>:8002
capath = /etc/pki/ca-trust/source
proxy-ntlm


curl -v https://www.kernel.org/pub/software/admin/A3Com/A3Com-0.2.3.tar.gz -# > kernel_gz.page  
*   Trying <IP>...
* TCP_NODELAY set
* Connected to <IP> (<IP>) port 8002 (#0)
* Establish HTTP proxy tunnel to www.kernel.org:443
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* Proxy auth using NTLM with user '<user>'
> CONNECT www.kernel.org:443 HTTP/1.1
> Host: www.kernel.org:443
> Proxy-Authorization: NTLM TlRMTVNTUAABAAAABoIIAAAAAAAAAAAAAAAAAAAAAAA=
> User-Agent: curl/7.51.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM TlRMTVNTUAACAAAABwAHADgAAAAGgokAoheStSQVTm4AAAAAAAAAALQAtAA/AAAABQCTCAAAAA9XSU5ET1dTAgAOAFcASQBOAEQATwBXAFMAAQAcAEIATABVAEUAQwBPAEEAVABQAFIATwBYAFkAQQAEACwAVwBJAE4ARABPAFcAUwAuAE4AWQBDAC4ASABSAEEALgBOAFkAQwBOAEUAVAADAEoAYgBsAHUAZQBjAG8AYQB0AHAAcgBvAHgAeQBhAC4AdwBpAG4AZABvAHcAcwAuAG4AeQBjAC4AaAByAGEALgBuAHkAYwBuAGUAdAAAAAAA
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
< Content-Length: 866
< 
* Ignore 866 bytes of response-body
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to www.kernel.org:443
* Proxy auth using NTLM with user '<user>'
> CONNECT www.kernel.org:443 HTTP/1.1
> Host: www.kernel.org:443
> Proxy-Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEAAAADkAOQAWAAAAAAAAAA8AQAACQAJADwBAAAGAAYARQEAAAAAAAAAAAAABoKJAE/NfbwMI+VuOztMa+99MIjEGCsGKGnapAUfnoVrNknsd5PShAd1XAcBAQAAAAAAAIC8M7CCv9IBxBgrBihp2qQAAAAAAgAOAFcASQBOAEQATwBXAFMAAQAcAEIATABVAEUAQwBPAEEAVABQAFIATwBYAFkAQQAEACwAVwBJAE4ARABPAFcAUwAuAE4AWQBDAC4ASABSAEEALgBOAFkAQwBOAEUAVAADAEoAYgBsAHUAZQBjAG8AYQB0AHAAcgBvAHgAeQBhAC4AdwBpAG4AZABvAHcAcwAuAG4AeQBjAC4AaAByAGEALgBuAHkAYwBuAGUAdAAAAAAAAAAAAGN6eW1hODg2MW5ldGRldg==
> User-Agent: curl/7.51.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied OK to CONNECT request
* failed to load '/etc/pki/ca-trust/source/README' from CURLOPT_CAPATH
* failed to load '/etc/pki/ca-trust/source/anchors' from CURLOPT_CAPATH
* failed to load '/etc/pki/ca-trust/source/blacklist' from CURLOPT_CAPATH
* failed to load '/etc/pki/ca-trust/source/ca-bundle.legacy.crt' from CURLOPT_CAPATH
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: /etc/pki/ca-trust/source
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=kernel.org,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated
* 	start date: Oct 11 00:00:00 2016 GMT
* 	expire date: Oct 11 23:59:59 2019 GMT
* 	common name: kernel.org
* 	issuer: E=<user>@<blah>.nyc.gov,CN=NYCHRA-PROXY-A,OU=MIS,O=NYCHRA,L=Brooklyn,ST=NY,C=US
> GET /pub/software/admin/A3Com/A3Com-0.2.3.tar.gz HTTP/1.1
> Host: www.kernel.org
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 26 Apr 2017 18:32:04 GMT
< Content-Type: application/x-gzip
< Content-Length: 25267
< Last-Modified: Fri, 04 Jun 1999 01:47:51 GMT
< Accept-Ranges: bytes
< Strict-Transport-Security: max-age=15768000
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< X-Frontend: packetnet-newark
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
< Age: 85590
< 
{ [6576 bytes data]
######################################################################## 100.0%* Curl_http_done: called premature == 0
* Connection #0 to host 10.253.10.212 left intact

also, 
if i disable ssl verification in dnf.conf ( #sslverify ) and after whitelisting the ip with curl, dnf would work fine.

This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

(In reply to andrew from comment #8)
> also, 
> if i disable ssl verification in dnf.conf ( #sslverify ) and after
> whitelisting the ip with curl, dnf would work fine.

Your issue with SSL seems to be a SSL intercepting proxy.  Install your proxy's CA's key chain and that issue should go way.  So you're still left with curl no being able to deal with NTLM auth.

This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

I get this error after updating from f28 to f29. dnf update was working fine with f28 but does not with the same settings on f29.

This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Setting proxy_auth_method, proxy, proxy_username and proxy_password in /etc/dnf/dnf.conf seems to work on Fedora 29 (dnf-4.2.5-1.fc29.noarch), however keeping those in world-readable file is insecure.

librepo-1.9.6-2.fc29.x86_64

Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.