Bug 1219199

Could you please provide us a tcp dump of client-proxy communication?

Created attachment 1023265 [details]
DNF tcpdump

Created attachment 1023271 [details]
yum tcpdump

yum tcpdump for comparison.
It seems that yum tries other authentication methods besides 'basic'.
This proxy server only accepts NTLM auth.

Updated to 

dnf   1.0.1-2.fc22

on Fedora 22 and the problem still exists.

Is there a known work around for dnf behind proxies for auth methods other than "basic" ?

Hi Tomas, 

does librepo support NTLM auth?

Hi Michal,

yes, it does. But option LRO_PROXYAUTH must be enabled.
Otherwise only basic auth method is available.

(In reply to Jeff from comment #4)

> Is there a known work around for dnf behind proxies for auth methods other
> than "basic" ?

My workaround for the moment was:

# yum-deprecated install ...

Just set the proxy credentials like always in /etc/yum.conf

(In reply to Bruno de Paula Larini from comment #7)
> (In reply to Jeff from comment #4)
> 
> > Is there a known work around for dnf behind proxies for auth methods other
> > than "basic" ?
> 
> My workaround for the moment was:
> 
> # yum-deprecated install ...
> 
> Just set the proxy credentials like always in /etc/yum.conf

Perfect, thanks.

(In reply to Tomas Mlcoch from comment #6)
> Hi Michal,
> 
> yes, it does. But option LRO_PROXYAUTH must be enabled.
> Otherwise only basic auth method is available.

Hello

I did the above in the Handle class definition __init__ with

self.setopt(librepo.LRO_PROXYAUTH, True)

and now dnf authenticates.

Does anyone know the anticipated permanent fix ?

Thanks

Isn't there a flag in configure line to enable it, I wonder?

Nevermind, it's a python script duh...

(In reply to Jeff from comment #9)
> (In reply to Tomas Mlcoch from comment #6)
> > Hi Michal,
> > 
> > yes, it does. But option LRO_PROXYAUTH must be enabled.
> > Otherwise only basic auth method is available.
> 
> Hello
> 
> I did the above in the Handle class definition __init__ with
> 
> self.setopt(librepo.LRO_PROXYAUTH, True)
> 
> and now dnf authenticates.
> 
> Does anyone know the anticipated permanent fix ?
> 
> Thanks

Hi Jeff,
I have a same problem.
Were is .py script with flag self.setopt(librepo.LRO_PROXYAUTH, True) ?
I can't find it.

You can try following scratch build and provide feedback regarding NTLM auth - http://koji.fedoraproject.org/koji/taskinfo?taskID=10280867

(In reply to Timofey from comment #12)
> (In reply to Jeff from comment #9)
> > (In reply to Tomas Mlcoch from comment #6)
> > > Hi Michal,
> > > 
> > > yes, it does. But option LRO_PROXYAUTH must be enabled.
> > > Otherwise only basic auth method is available.
> > 
> > Hello
> > 
> > I did the above in the Handle class definition __init__ with
> > 
> > self.setopt(librepo.LRO_PROXYAUTH, True)
> > 
> > and now dnf authenticates.
> > 
> > Does anyone know the anticipated permanent fix ?
> > 
> > Thanks
> 
> Hi Jeff,
> I have a same problem.
> Were is .py script with flag self.setopt(librepo.LRO_PROXYAUTH, True) ?
> I can't find it.


Sorry for the late reply : My gmail sent the rehat email to "Promotions" tag and I didnt see it.

I added it manually in

/usr/lib/python2.7/site-packages/dnf/repo.py

in the Handle class __init__ def.

It was just to check if it was indeed the problem.

(In reply to Jeff from comment #14)
> (In reply to Timofey from comment #12)
> > (In reply to Jeff from comment #9)
> > > (In reply to Tomas Mlcoch from comment #6)
> > > > Hi Michal,
> > > > 
> > > > yes, it does. But option LRO_PROXYAUTH must be enabled.
> > > > Otherwise only basic auth method is available.
> > > 
> > > Hello
> > > 
> > > I did the above in the Handle class definition __init__ with
> > > 
> > > self.setopt(librepo.LRO_PROXYAUTH, True)
> > > 
> > > and now dnf authenticates.
> > > 
> > > Does anyone know the anticipated permanent fix ?
> > > 
> > > Thanks
> > 
> > Hi Jeff,
> > I have a same problem.
> > Were is .py script with flag self.setopt(librepo.LRO_PROXYAUTH, True) ?
> > I can't find it.
> 
> 
> Sorry for the late reply : My gmail sent the rehat email to "Promotions" tag
> and I didnt see it.
> 
> I added it manually in
> 
> /usr/lib/python2.7/site-packages/dnf/repo.py
> 
> in the Handle class __init__ def.
> 
> It was just to check if it was indeed the problem.

Thanks a lot! It works!

https://github.com/rpm-software-management/dnf/pull/312

Thanks :-)

dnf-1.0.2-2.fc22,hawkey-0.5.9-2.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/dnf-1.0.2-2.fc22,hawkey-0.5.9-2.fc22

Which proxy system are you guys behind?
It still fails for me with the following error message, when authenticating against MS Forefront TMG 2010 (not 2012 as I said before), using the proposed 1.0.2-2 version on Fedora 22.

Error: Failed to synchronize cache for repo 'fedora' from 'https://mirrors.fedoraproject.org/metalink?repo=fedora-22&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (56): Failure when receiving data from the peer for https://mirrors.fedoraproject.org/metalink?repo=fedora-22&arch=x86_64 [Invalid file descriptor]

I'll attach the tcpdump as soon as possible.

Created attachment 1055315 [details]
dnf-1.0.2-2.fc22 tcpdump

As promised.

I am new at grabbing these updates (to test)

What am I doing wrong if i cant see this update pending, with a "dnf update" even after enabling fedora-updates-testing repository ?

The update wasn't pushed yet, but you can get them from the link provided in comment 17.

Thanks. I installed them.

From what I can see in your pcap your proxy offers 

Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM

where as mine only offers NTLM and Basic
Proxy-Authenticate: NTLM
Proxy-Authenticate: BASIC realm="Company Group Proxy"

I don't see a subsequent request with the auth headers in your pcap either.

Is your Kerberos setup ? 

I have attached a pcap showing the NTLM auth.

Created attachment 1056691 [details]
pcap showing NTLM challenge

Yes, Kerberos is also allowed but I was expecting NTLM would work as it did with yum.
By the looks of your tcpdump the updated release didn't work for you also, did it?

Sorry, I didn't post the complete capture.

I just posted the first NTLM request response headers

I will post the complete one tomorrow at work. I am not behind a proxy now.

But yes, it worked

Attached is a screen shot showing the interaction.

I cannot attach the actual pcap because I cannot anonymize certain server names

I could mail it to you if it would help but the screenshot shows the successful auth.

Created attachment 1057291 [details]
Screenshot showing ntlm auth

Hi Jeff, I was thinking about what you pointed before: the order of the authentication methods offered by the proxy server. Your proxy server offers NTLM first so dnf happily accepts it, whereas MS TMG offers first Negotiate and maybe dnf can't "negotiate" so it gives up. I can't read the code so I'm just supposing things. Could someone confirm that?

Yeah.

From what I can see librepo uses libcurl to do the calls, so I tried enabling the debugging to see what options it was passing via curl_easy_setopt.
Couldn't see anything obvious.

Out of interest, are you able to download something with curl using Kerberos auth ?

I will check further this side.

According to 

http://curl.haxx.se/libcurl/c/CURLOPT_HTTPAUTH.html
and
http://curl.haxx.se/libcurl/c/CURLOPT_PROXYAUTH.html

there is a CURLAUTH_NEGOTIATE option.

From the librepo code 

 case LRO_PROXYAUTH:
        if (va_arg(arg, long) == 1)
            c_rc = curl_easy_setopt(c_h, CURLOPT_PROXYAUTH, CURLAUTH_ANY);
        else
            c_rc = curl_easy_setopt(c_h, CURLOPT_PROXYAUTH, CURLAUTH_BASIC);
        break;

it looks like librepo proxy auth is either set to "basic" or "any" which translate to CURLAUTH_BASIC and CURLAUTH_ANY.

From the libcurl docs :
For convenience, you can use the 'CURLAUTH_ANY' define (instead of a list with specific types) which allows libcurl to use whatever method it wants.

When asking for multiple types, libcurl will pick the available one it considers "best" in its own internal order of preference. 

Not sure what behaviour this would lead to.

Well, I tried running:
$ curl --proxy http://proxyaddress:8080 --proxy-user myuser:mypassword --proxy-negotiate http://mirror.provider.org/package.rpm >> package.rpm

results in the exact same 407 error as dnf.

Switching --proxy-negotiate to --proxy-ntlm works and I can download the package via proxy.

Passing any other curl auth method (even --proxy-anyauth) results in 407.

Hello Bruno

Does your machine authenticate at login with Kerberos ?

I just want to validate that the kerberos client is setup ok.

Yes, it is properly set up and Windows stations authenticate using Kerberos, but I've never dealt with it when integrating third-party software with Active Directory or other MS software, only NTLM (yum, samba, freeradius). But I could run any test over here using it if necessary.

Ok, sorry. I am not a fundi on Kerberos.

Have you tried the windows domain name in front of your username ?

Yes, I've tried:

DOMAIN\\user.name
DOMAIN\user.name
"DOMAIN\user.name"
user.name

The test password contains no special characters.

I've tried even using space instead of "=" in dnf.conf.

But I think it would return a 403 if it were anything related to the credentials.

I'm almost convinced it is something with dnf not being able to deal/accept the Negotiate auth proposed by the proxy server.

Package dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnf-1.0.2-3.fc22 hawkey-0.5.9-3.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-12577/dnf-1.0.2-3.fc22,hawkey-0.5.9-3.fc22
then log in and leave karma (feedback).

Hello Bruno

Can you rebuild librepo with libcurl debug enabled and then try? 

That is the only thing I can think of that will tell
us what decisions libcurl is making around the negotiate header.

I rebuilt librepo with libcurl debug enabled.

Attached is the output for the start of a NTLM challenge just to show the verbosity and actions that libcurl is performing.

It is very verbose and if the same binary is run with your setup, it should give you whats happening.

Created attachment 1058837 [details]
Example of libcurl verbose logging

Hi Jeff.
I was wondering how to enable librepo debugging on build. Is something to add to the spec file?

Additionally version 1.0.2-3 didn't help, the "invalid file descriptor" error persists.

Thanks.

Hello Bruno

Enabling the debug mode on librepo was not enough (cmake -DCMAKE_BUILD_TYPE="DEBUG" ..). This wont give you the debug information that libcurl gives.

I enabled debugging on the libcurl component in the librepo code by adding 

 curl_easy_setopt(h, CURLOPT_VERBOSE, 1);

in handle.c (in the librepo directory of the checkout)

CURL *
lr_get_curl_handle()
{
    CURL *h;

    lr_global_init();

    h = curl_easy_init();
    curl_easy_setopt(h, CURLOPT_FOLLOWLOCATION, 1);
    curl_easy_setopt(h, CURLOPT_MAXREDIRS, 6);
    curl_easy_setopt(h, CURLOPT_CONNECTTIMEOUT, LRO_CONNECTTIMEOUT_DEFAULT);
    curl_easy_setopt(h, CURLOPT_LOW_SPEED_TIME, LRO_LOWSPEEDTIME_DEFAULT);
    curl_easy_setopt(h, CURLOPT_LOW_SPEED_LIMIT, LRO_LOWSPEEDLIMIT_DEFAULT);
    curl_easy_setopt(h, CURLOPT_SSL_VERIFYHOST, 2);
    curl_easy_setopt(h, CURLOPT_SSL_VERIFYPEER, 1);
    curl_easy_setopt(h, CURLOPT_VERBOSE, 1);
    return h;
}

Then just ran the make and make install.

Verify the "prefix" too
I just softlinked as follows after the build

lrwxrwxrwx. 1 root root 29 Aug  3 08:56 /usr/lib64/librepo.so.0 -> /usr/local/lib64/librepo.so.0 

Hope that helps

Okay, got it.
As suspected, it seems to be related to the negotiate auth proposed by the proxy server.
Give it a look and tell us what you think.
The message in portuguese does not contain anything relevant.

Created attachment 1059111 [details]
DNF proxy auth against MS Forefront TMG

(In reply to Bruno Larini from comment #34)
> Yes, it is properly set up and Windows stations authenticate using Kerberos,
> but I've never dealt with it when integrating third-party software with
> Active Directory or other MS software, only NTLM (yum, samba, freeradius).
> But I could run any test over here using it if necessary.

I have to just clear this up, please bear with me and not see this as a repeat question or insult :-)

What I meant by comment 33 was that 
Is the workstation/server that dnf is running on had the Kerberos client properly configured (krb5.conf etc)?

The reason I ask is because it looks like libcurl is happy with the negotiation and decided to try Kerberos because it is seen as the "best" option.

"gss_init_sec_context() failed: : SPNEGO cannot find mechanisms to negotiate

But according to 

http://stackoverflow.com/questions/26064157/libcurl-negotiate-fallback-to-ntlm

libcurl will NOT fallback to NTLM if Kerberos fails.

I would just like to make sure that the Kerberos client on the machine that libcurl is doing the call from is setup correctly.
The above log line (from your attachment) would indicate not.

Are you able to authenticate (Kerberos) via the command line using kinit ?

eg kinit -p youruser

I see, sorry for the misunderstood.

I've added the Fedora 22 machine to AD domain by properly configuring krb5.conf and Samba, and now it can list and login domain users.

I've opened a session with a domain user (member of wheel) and ran: sudo dnf install vim
The output was exactly the same as before.

Then I tried: su -c 'dnf install vim'
and noticed it managed to negotiate, but still failed with an "Unknown error".

Still, I think having to add the Fedora machine into AD and configure PAM or SSSD just to use dnf would be too much work. There must be an easier way like it was with yum and NTLM (even though I don't know what to suggest).

Created attachment 1059506 [details]
DNF proxy auth against MS Forefront TMG with Kerberos

Added the Fedora machine into Active Directory then ran: su -c 'dnf install ...'

Yeah ok, agreed.

The only way to do that neatly is to improve librepo's LRO_PROXYAUTH option from a boolean behavior to one that specifies a preferred auth type or types (Similar to the bitmask implementation libcurl has for its CURLOPT_PROXYAUTH option)

Not just between "CURLAUTH_BASIC" and "CURLAUTH_ANY" as it is now.

Just out of curiosity I replaced CURLAUTH_ANY by CURLAUTH_NTLM in the handle.c file and the proxy auth worked! It read the configuration in dnf.conf (instead of having to add to AD) and managed to download and install the package via proxy server.

Of course I know this is nothing near the final solution, but maybe there is some clever (cleaner) way to add it without breaking what is already working?

Yep, it would. This was proven in comment 32.

The problem is that when CURLAUTH_ANY is specified, curl will not default to NTLM if the Kerberos fails. The opinion is that the client app should explicitly set the auth option to NTLM.

I had a look at what they do in yum, because it uses curl too.

They use a package called urlgrabber and, if you read the entire interaction, they were faced with the same problem.

https://bugzilla.redhat.com/show_bug.cgi?id=769254

They way it was resolved is that they disabled Kerberos auth (enabled all but Kerberos) when using curl. (Comment 23)

http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=commit;h=0f8630eee377dc6c5470614f85d514f5f6f12067

Should the same approach be taken in librepo with dnf ?

If yes, I could attempt a patch.

Ok, I think it's right time to reassign this to librepo. Thank you all for the exhausting identification of actual problem.

dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

https://github.com/rpm-software-management/librepo/commit/bfc05df2f74ad7038977ce4c6b1fd367ec1da430