Bug 1219199 - Problem when using proxy authentication in dnf.conf
Problem when using proxy authentication in dnf.conf
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: librepo (Show other bugs)
21
x86_64 Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomas Mlcoch
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-05-06 15:58 EDT by Bruno Larini
Modified: 2015-09-24 09:14 EDT (History)
10 users (show)

See Also:
Fixed In Version: dnf-1.0.2-3.fc22
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-08-10 22:08:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
DNF tcpdump (2.08 KB, application/octet-stream)
2015-05-07 16:55 EDT, Bruno Larini
no flags Details
yum tcpdump (16.96 KB, application/octet-stream)
2015-05-07 17:20 EDT, Bruno Larini
no flags Details
dnf-1.0.2-2.fc22 tcpdump (4.16 KB, application/octet-stream)
2015-07-23 08:49 EDT, Bruno Larini
no flags Details
pcap showing NTLM challenge (3.00 KB, application/octet-stream)
2015-07-27 13:46 EDT, Jeff
no flags Details
Screenshot showing ntlm auth (292.68 KB, image/png)
2015-07-29 06:29 EDT, Jeff
no flags Details
Example of libcurl verbose logging (2.53 KB, text/plain)
2015-08-03 14:00 EDT, Jeff
no flags Details
DNF proxy auth against MS Forefront TMG (2.32 KB, text/plain)
2015-08-04 10:12 EDT, Bruno Larini
no flags Details
DNF proxy auth against MS Forefront TMG with Kerberos (3.40 KB, text/plain)
2015-08-05 11:27 EDT, Bruno Larini
no flags Details

  None (edit)
Description Bruno Larini 2015-05-06 15:58:59 EDT
Description of problem:

DNF gets a error message when using proxy configuration with authentication. The proxy server returns a 407 error (proxy authentication required), even though the username and password were specified as told in the DNF docs.
yum does work with the same proxy server when configured to use it.


Version-Release number of selected component:
Fedora Server 21
Kernel: 3.17.4-301.fc21.x86_64
DNF version: 0.6.4


Steps to Reproduce:
1. Add the following three lines to /etc/dnf/dnf.conf in [main] section:

proxy=http://192.168.0.1:8080
proxy_username=proxy.user
proxy_password=myproxypassword


2. Run an install/search command:

[root@fedora ~]# dnf install vim


3. Get error message.


Actual results:

Error: Failed to synchronize cache for repo 'fedora' from 'https://mirrors.fedoraproject.org/metalink?repo=fedora-21&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=fedora-21&arch=x86_64 [Received HTTP code 407 from proxy after CONNECT]


Expected results:

Receive the metadata info, download and install package.


Additional info:
- The proxy server runs Microsoft Forefront TMG 2012.
- Already tried switching the https:// prefix in the .repo files to http://, but the error persists.
Comment 1 Michal Luscon 2015-05-07 03:42:57 EDT
Could you please provide us a tcp dump of client-proxy communication?
Comment 2 Bruno Larini 2015-05-07 16:55:11 EDT
Created attachment 1023265 [details]
DNF tcpdump
Comment 3 Bruno Larini 2015-05-07 17:20:53 EDT
Created attachment 1023271 [details]
yum tcpdump

yum tcpdump for comparison.
It seems that yum tries other authentication methods besides 'basic'.
This proxy server only accepts NTLM auth.
Comment 4 Jeff 2015-07-01 02:41:29 EDT
Updated to 

dnf   1.0.1-2.fc22

on Fedora 22 and the problem still exists.

Is there a known work around for dnf behind proxies for auth methods other than "basic" ?
Comment 5 Michal Luscon 2015-07-01 04:02:25 EDT
Hi Tomas, 

does librepo support NTLM auth?
Comment 6 Tomas Mlcoch 2015-07-01 04:22:19 EDT
Hi Michal,

yes, it does. But option LRO_PROXYAUTH must be enabled.
Otherwise only basic auth method is available.
Comment 7 Bruno Larini 2015-07-01 08:33:42 EDT
(In reply to Jeff from comment #4)

> Is there a known work around for dnf behind proxies for auth methods other
> than "basic" ?

My workaround for the moment was:

# yum-deprecated install ...

Just set the proxy credentials like always in /etc/yum.conf
Comment 8 Jeff 2015-07-01 09:09:44 EDT
(In reply to Bruno de Paula Larini from comment #7)
> (In reply to Jeff from comment #4)
> 
> > Is there a known work around for dnf behind proxies for auth methods other
> > than "basic" ?
> 
> My workaround for the moment was:
> 
> # yum-deprecated install ...
> 
> Just set the proxy credentials like always in /etc/yum.conf

Perfect, thanks.
Comment 9 Jeff 2015-07-01 09:14:46 EDT
(In reply to Tomas Mlcoch from comment #6)
> Hi Michal,
> 
> yes, it does. But option LRO_PROXYAUTH must be enabled.
> Otherwise only basic auth method is available.

Hello

I did the above in the Handle class definition __init__ with

self.setopt(librepo.LRO_PROXYAUTH, True)

and now dnf authenticates.

Does anyone know the anticipated permanent fix ?

Thanks
Comment 10 Bruno Larini 2015-07-01 15:11:14 EDT
Isn't there a flag in configure line to enable it, I wonder?
Comment 11 Bruno Larini 2015-07-01 15:16:25 EDT
Nevermind, it's a python script duh...
Comment 12 Timofey 2015-07-03 05:25:50 EDT
(In reply to Jeff from comment #9)
> (In reply to Tomas Mlcoch from comment #6)
> > Hi Michal,
> > 
> > yes, it does. But option LRO_PROXYAUTH must be enabled.
> > Otherwise only basic auth method is available.
> 
> Hello
> 
> I did the above in the Handle class definition __init__ with
> 
> self.setopt(librepo.LRO_PROXYAUTH, True)
> 
> and now dnf authenticates.
> 
> Does anyone know the anticipated permanent fix ?
> 
> Thanks

Hi Jeff,
I have a same problem.
Were is .py script with flag self.setopt(librepo.LRO_PROXYAUTH, True) ?
I can't find it.
Comment 13 Michal Luscon 2015-07-03 07:26:53 EDT
You can try following scratch build and provide feedback regarding NTLM auth - http://koji.fedoraproject.org/koji/taskinfo?taskID=10280867
Comment 14 Jeff 2015-07-13 15:09:29 EDT
(In reply to Timofey from comment #12)
> (In reply to Jeff from comment #9)
> > (In reply to Tomas Mlcoch from comment #6)
> > > Hi Michal,
> > > 
> > > yes, it does. But option LRO_PROXYAUTH must be enabled.
> > > Otherwise only basic auth method is available.
> > 
> > Hello
> > 
> > I did the above in the Handle class definition __init__ with
> > 
> > self.setopt(librepo.LRO_PROXYAUTH, True)
> > 
> > and now dnf authenticates.
> > 
> > Does anyone know the anticipated permanent fix ?
> > 
> > Thanks
> 
> Hi Jeff,
> I have a same problem.
> Were is .py script with flag self.setopt(librepo.LRO_PROXYAUTH, True) ?
> I can't find it.


Sorry for the late reply : My gmail sent the rehat email to "Promotions" tag and I didnt see it.

I added it manually in

/usr/lib/python2.7/site-packages/dnf/repo.py

in the Handle class __init__ def.

It was just to check if it was indeed the problem.
Comment 15 Timofey 2015-07-14 09:46:33 EDT
(In reply to Jeff from comment #14)
> (In reply to Timofey from comment #12)
> > (In reply to Jeff from comment #9)
> > > (In reply to Tomas Mlcoch from comment #6)
> > > > Hi Michal,
> > > > 
> > > > yes, it does. But option LRO_PROXYAUTH must be enabled.
> > > > Otherwise only basic auth method is available.
> > > 
> > > Hello
> > > 
> > > I did the above in the Handle class definition __init__ with
> > > 
> > > self.setopt(librepo.LRO_PROXYAUTH, True)
> > > 
> > > and now dnf authenticates.
> > > 
> > > Does anyone know the anticipated permanent fix ?
> > > 
> > > Thanks
> > 
> > Hi Jeff,
> > I have a same problem.
> > Were is .py script with flag self.setopt(librepo.LRO_PROXYAUTH, True) ?
> > I can't find it.
> 
> 
> Sorry for the late reply : My gmail sent the rehat email to "Promotions" tag
> and I didnt see it.
> 
> I added it manually in
> 
> /usr/lib/python2.7/site-packages/dnf/repo.py
> 
> in the Handle class __init__ def.
> 
> It was just to check if it was indeed the problem.

Thanks a lot! It works!
Comment 17 Fedora Update System 2015-07-22 04:25:18 EDT
dnf-1.0.2-2.fc22,hawkey-0.5.9-2.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/dnf-1.0.2-2.fc22,hawkey-0.5.9-2.fc22
Comment 18 Bruno Larini 2015-07-22 17:09:38 EDT
Which proxy system are you guys behind?
It still fails for me with the following error message, when authenticating against MS Forefront TMG 2010 (not 2012 as I said before), using the proposed 1.0.2-2 version on Fedora 22.

Error: Failed to synchronize cache for repo 'fedora' from 'https://mirrors.fedoraproject.org/metalink?repo=fedora-22&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=fedora-22&arch=x86_64 [Invalid file descriptor]

I'll attach the tcpdump as soon as possible.
Comment 19 Bruno Larini 2015-07-23 08:49:38 EDT
Created attachment 1055315 [details]
dnf-1.0.2-2.fc22 tcpdump

As promised.
Comment 20 Jeff 2015-07-25 15:13:10 EDT
I am new at grabbing these updates (to test)

What am I doing wrong if i cant see this update pending, with a "dnf update" even after enabling fedora-updates-testing repository ?
Comment 21 Bruno Larini 2015-07-26 13:27:12 EDT
The update wasn't pushed yet, but you can get them from the link provided in comment 17.
Comment 22 Jeff 2015-07-27 13:42:57 EDT
Thanks. I installed them.

From what I can see in your pcap your proxy offers 

Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM

where as mine only offers NTLM and Basic
Proxy-Authenticate: NTLM
Proxy-Authenticate: BASIC realm="Company Group Proxy"

I don't see a subsequent request with the auth headers in your pcap either.

Is your Kerberos setup ? 

I have attached a pcap showing the NTLM auth.
Comment 23 Jeff 2015-07-27 13:46:07 EDT
Created attachment 1056691 [details]
pcap showing NTLM challenge
Comment 24 Bruno Larini 2015-07-27 14:20:06 EDT
Yes, Kerberos is also allowed but I was expecting NTLM would work as it did with yum.
By the looks of your tcpdump the updated release didn't work for you also, did it?
Comment 25 Jeff 2015-07-27 14:44:18 EDT
Sorry, I didn't post the complete capture.

I just posted the first NTLM request response headers

I will post the complete one tomorrow at work. I am not behind a proxy now.
Comment 26 Jeff 2015-07-27 14:49:27 EDT
But yes, it worked
Comment 27 Jeff 2015-07-29 06:27:36 EDT
Attached is a screen shot showing the interaction.

I cannot attach the actual pcap because I cannot anonymize certain server names

I could mail it to you if it would help but the screenshot shows the successful auth.
Comment 28 Jeff 2015-07-29 06:29:18 EDT
Created attachment 1057291 [details]
Screenshot showing ntlm auth
Comment 29 Bruno Larini 2015-07-29 08:09:43 EDT
Hi Jeff, I was thinking about what you pointed before: the order of the authentication methods offered by the proxy server. Your proxy server offers NTLM first so dnf happily accepts it, whereas MS TMG offers first Negotiate and maybe dnf can't "negotiate" so it gives up. I can't read the code so I'm just supposing things. Could someone confirm that?
Comment 30 Jeff 2015-07-29 08:27:37 EDT
Yeah.

From what I can see librepo uses libcurl to do the calls, so I tried enabling the debugging to see what options it was passing via curl_easy_setopt.
Couldn't see anything obvious.

Out of interest, are you able to download something with curl using Kerberos auth ?

I will check further this side.
Comment 31 Jeff 2015-07-29 09:03:52 EDT
According to 

http://curl.haxx.se/libcurl/c/CURLOPT_HTTPAUTH.html
and
http://curl.haxx.se/libcurl/c/CURLOPT_PROXYAUTH.html

there is a CURLAUTH_NEGOTIATE option.

From the librepo code 

 case LRO_PROXYAUTH:
        if (va_arg(arg, long) == 1)
            c_rc = curl_easy_setopt(c_h, CURLOPT_PROXYAUTH, CURLAUTH_ANY);
        else
            c_rc = curl_easy_setopt(c_h, CURLOPT_PROXYAUTH, CURLAUTH_BASIC);
        break;

it looks like librepo proxy auth is either set to "basic" or "any" which translate to CURLAUTH_BASIC and CURLAUTH_ANY.

From the libcurl docs :
For convenience, you can use the 'CURLAUTH_ANY' define (instead of a list with specific types) which allows libcurl to use whatever method it wants.

When asking for multiple types, libcurl will pick the available one it considers "best" in its own internal order of preference. 

Not sure what behaviour this would lead to.
Comment 32 Bruno Larini 2015-07-29 09:31:06 EDT
Well, I tried running:
$ curl --proxy http://proxyaddress:8080 --proxy-user myuser:mypassword --proxy-negotiate http://mirror.provider.org/package.rpm >> package.rpm

results in the exact same 407 error as dnf.

Switching --proxy-negotiate to --proxy-ntlm works and I can download the package via proxy.

Passing any other curl auth method (even --proxy-anyauth) results in 407.
Comment 33 Jeff 2015-07-30 07:32:27 EDT
Hello Bruno

Does your machine authenticate at login with Kerberos ?

I just want to validate that the kerberos client is setup ok.
Comment 34 Bruno Larini 2015-07-30 08:25:32 EDT
Yes, it is properly set up and Windows stations authenticate using Kerberos, but I've never dealt with it when integrating third-party software with Active Directory or other MS software, only NTLM (yum, samba, freeradius). But I could run any test over here using it if necessary.
Comment 35 Jeff 2015-07-30 13:13:38 EDT
Ok, sorry. I am not a fundi on Kerberos.

Have you tried the windows domain name in front of your username ?
Comment 36 Bruno Larini 2015-07-30 15:27:04 EDT
Yes, I've tried:

DOMAIN\\user.name
DOMAIN\user.name
"DOMAIN\user.name"
user.name@domain.local

The test password contains no special characters.

I've tried even using space instead of "=" in dnf.conf.

But I think it would return a 403 if it were anything related to the credentials.

I'm almost convinced it is something with dnf not being able to deal/accept the Negotiate auth proposed by the proxy server.
Comment 37 Fedora Update System 2015-07-31 22:28:26 EDT
Package dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnf-1.0.2-3.fc22 hawkey-0.5.9-3.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-12577/dnf-1.0.2-3.fc22,hawkey-0.5.9-3.fc22
then log in and leave karma (feedback).
Comment 38 Jeff 2015-08-01 09:33:39 EDT
Hello Bruno

Can you rebuild librepo with libcurl debug enabled and then try? 

That is the only thing I can think of that will tell
us what decisions libcurl is making around the negotiate header.
Comment 39 Jeff 2015-08-03 13:58:06 EDT
I rebuilt librepo with libcurl debug enabled.

Attached is the output for the start of a NTLM challenge just to show the verbosity and actions that libcurl is performing.

It is very verbose and if the same binary is run with your setup, it should give you whats happening.
Comment 40 Jeff 2015-08-03 14:00:36 EDT
Created attachment 1058837 [details]
Example of libcurl verbose logging
Comment 41 Bruno Larini 2015-08-03 14:27:17 EDT
Hi Jeff.
I was wondering how to enable librepo debugging on build. Is something to add to the spec file?

Additionally version 1.0.2-3 didn't help, the "invalid file descriptor" error persists.

Thanks.
Comment 42 Jeff 2015-08-03 15:03:56 EDT
Hello Bruno

Enabling the debug mode on librepo was not enough (cmake -DCMAKE_BUILD_TYPE="DEBUG" ..). This wont give you the debug information that libcurl gives.

I enabled debugging on the libcurl component in the librepo code by adding 

 curl_easy_setopt(h, CURLOPT_VERBOSE, 1);

in handle.c (in the librepo directory of the checkout)

CURL *
lr_get_curl_handle()
{
    CURL *h;

    lr_global_init();

    h = curl_easy_init();
    curl_easy_setopt(h, CURLOPT_FOLLOWLOCATION, 1);
    curl_easy_setopt(h, CURLOPT_MAXREDIRS, 6);
    curl_easy_setopt(h, CURLOPT_CONNECTTIMEOUT, LRO_CONNECTTIMEOUT_DEFAULT);
    curl_easy_setopt(h, CURLOPT_LOW_SPEED_TIME, LRO_LOWSPEEDTIME_DEFAULT);
    curl_easy_setopt(h, CURLOPT_LOW_SPEED_LIMIT, LRO_LOWSPEEDLIMIT_DEFAULT);
    curl_easy_setopt(h, CURLOPT_SSL_VERIFYHOST, 2);
    curl_easy_setopt(h, CURLOPT_SSL_VERIFYPEER, 1);
    curl_easy_setopt(h, CURLOPT_VERBOSE, 1);
    return h;
}

Then just ran the make and make install.

Verify the "prefix" too
I just softlinked as follows after the build

lrwxrwxrwx. 1 root root 29 Aug  3 08:56 /usr/lib64/librepo.so.0 -> /usr/local/lib64/librepo.so.0 

Hope that helps
Comment 43 Bruno Larini 2015-08-04 10:10:58 EDT
Okay, got it.
As suspected, it seems to be related to the negotiate auth proposed by the proxy server.
Give it a look and tell us what you think.
The message in portuguese does not contain anything relevant.
Comment 44 Bruno Larini 2015-08-04 10:12:48 EDT
Created attachment 1059111 [details]
DNF proxy auth against MS Forefront TMG
Comment 45 Jeff 2015-08-05 07:07:14 EDT
(In reply to Bruno Larini from comment #34)
> Yes, it is properly set up and Windows stations authenticate using Kerberos,
> but I've never dealt with it when integrating third-party software with
> Active Directory or other MS software, only NTLM (yum, samba, freeradius).
> But I could run any test over here using it if necessary.

I have to just clear this up, please bear with me and not see this as a repeat question or insult :-)

What I meant by comment 33 was that 
Is the workstation/server that dnf is running on had the Kerberos client properly configured (krb5.conf etc)?

The reason I ask is because it looks like libcurl is happy with the negotiation and decided to try Kerberos because it is seen as the "best" option.

"gss_init_sec_context() failed: : SPNEGO cannot find mechanisms to negotiate

But according to 

http://stackoverflow.com/questions/26064157/libcurl-negotiate-fallback-to-ntlm

libcurl will NOT fallback to NTLM if Kerberos fails.

I would just like to make sure that the Kerberos client on the machine that libcurl is doing the call from is setup correctly.
The above log line (from your attachment) would indicate not.

Are you able to authenticate (Kerberos) via the command line using kinit ?

eg kinit -p youruser@domainname.com
Comment 46 Bruno Larini 2015-08-05 11:25:39 EDT
I see, sorry for the misunderstood.

I've added the Fedora 22 machine to AD domain by properly configuring krb5.conf and Samba, and now it can list and login domain users.

I've opened a session with a domain user (member of wheel) and ran: sudo dnf install vim
The output was exactly the same as before.

Then I tried: su -c 'dnf install vim'
and noticed it managed to negotiate, but still failed with an "Unknown error".

Still, I think having to add the Fedora machine into AD and configure PAM or SSSD just to use dnf would be too much work. There must be an easier way like it was with yum and NTLM (even though I don't know what to suggest).
Comment 47 Bruno Larini 2015-08-05 11:27:28 EDT
Created attachment 1059506 [details]
DNF proxy auth against MS Forefront TMG with Kerberos

Added the Fedora machine into Active Directory then ran: su -c 'dnf install ...'
Comment 48 Jeff 2015-08-05 16:24:39 EDT
Yeah ok, agreed.

The only way to do that neatly is to improve librepo's LRO_PROXYAUTH option from a boolean behavior to one that specifies a preferred auth type or types (Similar to the bitmask implementation libcurl has for its CURLOPT_PROXYAUTH option)

Not just between "CURLAUTH_BASIC" and "CURLAUTH_ANY" as it is now.
Comment 49 Bruno Larini 2015-08-05 17:29:59 EDT
Just out of curiosity I replaced CURLAUTH_ANY by CURLAUTH_NTLM in the handle.c file and the proxy auth worked! It read the configuration in dnf.conf (instead of having to add to AD) and managed to download and install the package via proxy server.

Of course I know this is nothing near the final solution, but maybe there is some clever (cleaner) way to add it without breaking what is already working?
Comment 50 Jeff 2015-08-06 02:06:37 EDT
Yep, it would. This was proven in comment 32.

The problem is that when CURLAUTH_ANY is specified, curl will not default to NTLM if the Kerberos fails. The opinion is that the client app should explicitly set the auth option to NTLM.

I had a look at what they do in yum, because it uses curl too.

They use a package called urlgrabber and, if you read the entire interaction, they were faced with the same problem.

https://bugzilla.redhat.com/show_bug.cgi?id=769254

They way it was resolved is that they disabled Kerberos auth (enabled all but Kerberos) when using curl. (Comment 23)

http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=commit;h=0f8630eee377dc6c5470614f85d514f5f6f12067

Should the same approach be taken in librepo with dnf ?

If yes, I could attempt a patch.
Comment 51 Michal Luscon 2015-08-06 03:43:21 EDT
Ok, I think it's right time to reassign this to librepo. Thank you all for the exhausting identification of actual problem.
Comment 52 Fedora Update System 2015-08-10 22:08:23 EDT
dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.