Bug 122304 - yum on x86_64 is fetching the wrong URL
yum on x86_64 is fetching the wrong URL
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: python (Show other bugs)
rawhide
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Jeremy Katz
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-05-02 18:00 EDT by Steven Pritchard
Modified: 2014-01-21 17:49 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-01-06 16:07:14 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
tethereal packet dump (1.27 KB, application/octet-stream)
2004-05-05 12:22 EDT, Simon Clifford
no flags Details

  None (edit)
Description Steven Pritchard 2004-05-02 18:00:47 EDT
Description of problem:
I have an entry in /etc/yum.conf to fetch packages from rawhide:

[rawhide]
name=Fedora Core Development
baseurl=http://ftp.kspei.com/pub/fedora/linux/core/development/x86_64

Running "yum update" results in this:

Gathering header information file(s) from server(s)
Server: Fedora Core 1.92 - x86_64 - Base
Server: Fedora Core Development
retrygrab() failed for:
 
http://ftp.kspei.com/pub/fedora/linux/core/development/x86_64/headers/header.info
  Executing failover method
failover: out of servers to try
Error getting file
http://ftp.kspei.com/pub/fedora/linux/core/development/x86_64/headers/header.info
[Errno 4] IOError: HTTP Error 400: Bad Request

Squid (running in transparent proxy mode) shows that the problem is a
bad URL:

1083535141.559      1 xxx.xxx.x.xx NONE/400 1619 GET
http://ftp.kspei..om:80/pub/fedora/linux/core/development/x86_64/headers/header.info
- NONE/- text/html

Note that it is trying to get "ftp.kspei..om" instead of "ftp.kspei.com".

Version-Release number of selected component (if applicable):
2.0.7-0.20040416

How reproducible:
Every time.

Steps to Reproduce:
1. Add an entry to /etc/yum.conf with a hostname in the URL.
2. yum update
  
Actual results:
Above error.

Expected results:
Successful download.

Additional info:
Changing the hostname to an IP address in the URL works around the
problem.

I'm assuming this is an x86_64-only problem, but I don't have any
other systems running FC2t3 to test with.
Comment 1 Seth Vidal 2004-05-04 22:19:13 EDT
I can't replicate this at all. Can you do the following:

run 
python /usr/share/yum/urlgrabber.py \
http://ftp.kspei.com/pub/fedora/linux/core/development/x86_64/headers/header.info
\  /tmp/header.info

(that's all one line)

that will run the module that yum uses to download things.

see if the error occurs there, too. If so it's either in urlgrabber
(doubtful) or it's something wonky in the build of python for x86_64.

Comment 2 Simon Clifford 2004-05-05 12:22:17 EDT
Created attachment 99994 [details]
tethereal packet dump
Comment 3 Simon Clifford 2004-05-05 12:23:16 EDT
Hi,

I'm having the same problem as Stephen. My symptoms are the same as
Stephen's, although I cannot use yum with *any* baseurl -- not even
the distributed one of:
baseurl=http://download.fedora.redhat.com/pub/fedora/linux/core/development/$basearch

I always get the error:
Server: Fedora Core 1.92 - Development Tree
retrygrab() failed for:
 
http://download.fedora.redhat.com/pub/fedora/linux/core/development/x86_64/headers/header.info
  Executing failover method
failover: out of servers to try
Error getting file
http://download.fedora.redhat.com/pub/fedora/linux/core/development/x86_64/headers/header.info
[Errno 4] IOError: HTTP Error 400: Bad Request

If I try:
python /usr/share/yum/urlgrabber.py
http://download.fedora.redhat.com/pub/fedora/linux/core/development/x86_64/headers/header.info
/tmp/header.info # all on one line

I get:

throttle: 1.0,  throttle bandwidth: 32768 B/s
[Errno 4] IOError: HTTP Error 400: Bad Request

Sniffing the packets shows that python is (as Stephen said) replacing
the '.com' in the Host header of the HTTP 1.1 request with '..om'.

I've attached a tethereal packet file of the http packets generated by
the urlgrabber command.
Comment 4 Thom May 2004-05-05 12:52:55 EDT
This is really just another me too, but I'm happy to try any further
debugging you need.
Comment 5 Seth Vidal 2004-05-05 13:25:21 EDT
this is happening before the connection is ever made - this is
happening during the name resolution afaict.

and since I'm only hearing about this from x86_64 it worries me for
that python  build.

Just to be sure - all of y'all are x86_64 fc2 - what ver of python?
rpm -q python should tell me what I need

also - are all of you behind proxies?
Comment 6 Seth Vidal 2004-05-06 02:08:03 EDT
ok - this is not yum - it's python on x86_64 in python 2.3.3
I found something that could be a problem on x86_64 for python:

try this on x86_64:
foo='www.redhat.com'
foo.encode("idna")
depending on your encoding that's set you'll get either the correct:
www.redhat.com
or
www.redhat..om

we look on line 6 of 
/usr/lib64/python2.3/encodings/idna.py at:
dots = re.compile(u"[\u002E\u3002\uFF0E\uFF61]")

that works great on x86 - so a little further down on line 153 you see:

labels = dots.split(input)

the input in this question is like the url above.

so try this bit of code on your own x86_64 python 2.3.3 system:

import re
dots = re.compile(u"[\u002E\u3002\uFF0E\uFF61]")
foo = 'www.redhat.com'
labels = dots.split(foo)

print labels

you'll find it is:
Hi,
 Troubleshooting this bug:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=122304

I found something that could be a problem on x86_64 for python:

try this on x86_64:
foo='www.redhat.com'
foo.encode("idna")
depending on your encoding that's set you'll get either the correct:
www.redhat.com
or
www.redhat..om

we look on line 6 of 
/usr/lib64/python2.3/encodings/idna.py at:
dots = re.compile(u"[\u002E\u3002\uFF0E\uFF61]")

that works great on x86 - so a little further down on line 153 you see:

labels = dots.split(input)

the input in this question is like the url above.

so try this bit of code on your own x86_64 python 2.3.3 system:

import re
dots = re.compile(u"[\u002E\u3002\uFF0E\uFF61]")
foo = 'www.redhat.com'
labels = dots.split(foo)

print labels

you'll find it is:
['www.redhat.', 'om']

while on x86 it is:

['www', 'redhat', 'com']

which is correct - 3 label sections from rfc 3490

so I went looking for the problem a little bit and found in _sre.c
    #if defined(MS_WIN64) || defined(__LP64__) || defined(_LP64)
        /* require smaller recursion limit for a number of 64-bit
platforms:
         * Win64 (MS_WIN64), Linux64 (__LP64__), Monterey (64-bit AIX)
(_LP64)
         */
        /* FIXME: maybe the limit should be 40000 / sizeof(void*) ? */
        #define USE_RECURSION_LIMIT 7500



I'm wondering if that FIXME is accurate - I've not tested the change
yet but it seems like a potential problem for regexes like this - or
more to the point anything using the HTTPHandler in python.

Can someone more experienced at python _sre internals take a look at this?

this will probably make up2date blow up and a number of python-based
network-accessing programs.
Comment 7 Seth Vidal 2004-05-06 02:36:46 EDT
found the bug filed, by misa@redhat.com on python's bug tracker

http://sourceforge.net/tracker/index.php?func=detail&aid=931848&group_id=5470&atid=105470

Should this be set to a blocker for FC2 b/c of other potentially
affected apps?

Comment 8 Seth Vidal 2004-05-06 14:51:33 EDT
--- idna.py~    2004-05-06 14:55:34.000000000 -0400
+++ idna.py     2004-05-06 14:55:34.000000000 -0400
@@ -3,7 +3,7 @@
 import stringprep, unicodedata, re, codecs
  
 # IDNA section 3.1
-dots = re.compile(u"[\u002E\u3002\uFF0E\uFF61]")
+dots = re.compile(u"[\u002E]")
  
 # IDNA section 5
 ace_prefix = "xn--"



that patch applied to:
/usr/lib64/python2.3/encodings/idna.py will "fix" this problem.
Comment 9 Mihai Ibanescu 2004-05-06 18:37:05 EDT
Patch applied in python-2.3.3-4
Comment 10 Bert de Bruijn 2004-08-18 09:02:58 EDT
I see the same behaviour on the alpha platform, using python-2.3.3-6.
with the proposed fix, things (yum, most importantly) work OK.
Comment 11 Mike Barnes 2004-08-28 22:12:15 EDT
Can confirm the python-2.3.3-lib64-regex patch included fixes this bug on the Alpha, but 
it's only included by the spec file if the lib path is set to "lib64". The alpha only uses "lib", 
since there was never a 32-bit version of the platform.

Is there a better way to probe for a 64-bit architecture in the spec file, other than looking 
to see if %{_lib} is set to lib64, or manually keeping a list and %ifarch-ing the thing?

The Alpha's making a comeback - I swear. :)
Comment 12 John Thacker 2006-10-30 09:48:07 EST
[This is a mass update sent to many bugs that missed earlier such messages due
to having their version set to a test version.]

This bug was originally filed against a version of Fedora Core which is no
longer supported, even for security updates.  Many changes have occured since
then.  Please retest this bug against a still supported version.  Note that FC3
and FC4 are supported by Fedora Legacy for security fixes only.  If
it still occurs on FC5 or FC6, please assign to the correct
version.  Otherwise, if this a security issue, please change the
product to Fedora Legacy.  Thanks, and we are sorry that we did not
get to this bug earlier.

This bug will be closed after a few weeks if no information is given indicating
that the bug is still present in a supported release.
Comment 13 Jeremy Katz 2007-01-06 16:07:14 EST
Added alpha to the arches this patch is applied for in CVS, will be in python >
2.5-7
Comment 14 Dave Malcolm 2010-01-29 18:12:13 EST
(In reply to comment #7)
> found the bug filed, by misa@redhat.com on python's bug tracker
> 
> http://sourceforge.net/tracker/index.php?func=detail&aid=931848&group_id=5470&atid=105470

For reference, this upstream bug URL is now at:
http://bugs.python.org/issue931848

(working through the downstream patches in our Python specfile)

Note You need to log in before you can comment on or make changes to this bug.