Bug 771204 - curl has trouble with IPV6
Summary: curl has trouble with IPV6
Status: CLOSED DUPLICATE of bug 841787
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: glibc
Version: 6.1
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: rc
: ---
Assignee: Jeff Law
QA Contact: qe-baseos-tools
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-01-02 13:04 UTC by Klaus Steinberger
Modified: 2016-11-24 12:44 UTC (History)
5 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2012-07-23 19:44:13 UTC


Attachments (Terms of Use)
nsswitch.conf (1.65 KB, text/plain)
2012-01-10 07:43 UTC, Klaus Steinberger
no flags Details
Not working case (options rotate in resolv.conf and only IPV6 DNS) (20.33 KB, text/plain)
2012-01-10 11:33 UTC, Klaus Steinberger
no flags Details
Working case (only IPV6 DNS, but no options line in resolv.conf) (52.27 KB, text/plain)
2012-01-10 11:34 UTC, Klaus Steinberger
no flags Details

Description Klaus Steinberger 2012-01-02 13:04:03 UTC
Description of problem:

curl has name resolving problems when only a IPV6 resolver is available
(or it hits a IPV6 resolver through a resolver rotate)

Version-Release number of selected component (if applicable):
curl-7.19.7-26.el6_1.2.x86_64


How reproducible:
every time curl uses a IPV6 resolver.

The problem was already reported and solved for Fedora 13 in Bug #548269 

Please fix this problem also for RHEL 6

Comment 2 Kamil Dudka 2012-01-02 13:21:22 UTC
(In reply to comment #0)
> curl has name resolving problems when only a IPV6 resolver is available
> (or it hits a IPV6 resolver through a resolver rotate)

What kind of problems?  I do not think that RHEL-6 curl does anything special in the terms of name resolving.  It uses the standard NSS service from glibc.  If it fails, it is most likely problem in your network configuration and I bet that other clients on your machine suffer from the same problems.

> The problem was already reported and solved for Fedora 13 in Bug #548269 
> 
> Please fix this problem also for RHEL 6

Bug #548269 was caused by using c-ares for name resolving.  This problem was specific to Fedora 13 and it has never reached any RHEL.

Comment 3 Klaus Steinberger 2012-01-09 12:57:15 UTC
Hi,

i did some tests now, and can prove the problem.

It happens only when "option rotate" is set inside /etc/resolv.conf

For example:

nameserver 2001:4ca0:4103:3::2
nameserver 2001:4ca0:4103:1::10
search physik.uni-muenchen.de 
options rotate


without "options rotate" it works very well, but with "options rotate" set and ipv6 DNS servers yum fails:


[root@z-sv-rheltst etc]# yum update
Loaded plugins: product-id, rhnplugin, security, subscription-manager
Updating certificate-based repositories.
Bei der Kommunikation mit RHN trat ein Fehler auf.
RHN Satellite oder RHN Classic Unterstützung wird deaktiviert.
Fehler beim Kommunizieren mit dem Server. Die Meldung war:
Unable to connect to the host and port specified
Setting up Update Process
No Packages marked for Update
[root@z-sv-rheltst etc]# 


The bug is probably not in yum, but somewhere in the deeper layers, as the very same problem happens with IBM Tivoli client, but ping/ping6 works as expected.

Comment 4 Kamil Dudka 2012-01-09 13:36:12 UTC
This does not show me any problem in libcurl.  What does 'getent ahosts' say about the server you are trying to connect?

Comment 5 Klaus Steinberger 2012-01-09 14:11:57 UTC
getent ahosts:
[root@z-sv-rheltst ~]# getent ahosts
127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
127.0.0.1       localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@z-sv-rheltst ~]#

Id did some tracing with strace. I did the following:

yum clean all
strace yum update 2> tracefile

in the working case (no "options rotate"):

munmap(0x7faee7497000, 4096)            = 0
socket(PF_INET6, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 10
connect(10, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
gettimeofday({1326117741, 811677}, NULL) = 0
poll([{fd=10, events=POLLOUT}], 1, 0)   = 1 ([{fd=10, revents=POLLOUT}])
sendto(10, "\316f\1\0\0\1\0\0\0\0\0\0\6xmlrpc\3rhn\6redhat\3c"..., 39, MSG_NOSIGNAL, NULL, 0) = 39
poll([{fd=10, events=POLLIN}], 1, 5000) = 1 ([{fd=10, revents=POLLIN}])
ioctl(10, FIONREAD, [279])              = 0
recvfrom(10, "\316f\201\200\0\1\0\1\0\r\0\0\6xmlrpc\3rhn\6redhat\3c"..., 1024, 0, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 279
close(10)                               = 0


In the non working case (with "option rotate"):


munmap(0x7f5917a6b000, 4096)            = 0
socket(PF_INET6, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 10
connect(10, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
gettimeofday({1326117821, 673407}, NULL) = 0
poll([{fd=10, events=POLLOUT}], 1, 0)   = 1 ([{fd=10, revents=POLLOUT}])
sendto(10, "v\220\1\0\0\1\0\0\0\0\0\0\6xmlrpc\3rhn\6redhat\3c"..., 62, MSG_NOSIGNAL, NULL, 0) = 62
poll([{fd=10, events=POLLIN}], 1, 5000) = 1 ([{fd=10, revents=POLLIN}])
ioctl(10, FIONREAD, [117])              = 0
recvfrom(10, "v\220\205\203\0\1\0\0\0\1\0\0\6xmlrpc\3rhn\6redhat\3c"..., 1024, 0, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 117
close(10)                               = 0
close(9)                                = 0
gettimeofday({1326117821, 674909}, NULL) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
open("/var/log/up2date", O_WRONLY|O_CREAT|O_APPEND, 0666) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=5033, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5917a6b000
fstat(9, {st_mode=S_IFREG|0644, st_size=5033, ...}) = 0
lseek(9, 5033, SEEK_SET)                = 5033
fstat(9, {st_mode=S_IFREG|0644, st_size=5033, ...}) = 0
write(9, "[Mon Jan  9 15:03:41 2012] up2da"..., 121) = 121
close(9)                                = 0
munmap(0x7f5917a6b000, 4096)            = 0
gettimeofday({1326117821, 675925}, NULL) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
open("/var/log/up2date", O_WRONLY|O_CREAT|O_APPEND, 0666) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=5154, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5917a6b000
fstat(9, {st_mode=S_IFREG|0644, st_size=5154, ...}) = 0
lseek(9, 5154, SEEK_SET)                = 5154
fstat(9, {st_mode=S_IFREG|0644, st_size=5154, ...}) = 0
write(9, "[Mon Jan  9 15:03:41 2012] up2da"..., 143) = 143
close(9)                                = 0
munmap(0x7f5917a6b000, 4096)            = 0
gettimeofday({1326117821, 676649}, NULL) = 0
write(2, "Bei der Kommunikation mit RHN tr"..., 223Bei der Kommunikation mit RHN trat ein Fehler auf.


Looks like the resolver library didn't give back an IP Address if "options  rotate" was set ....

Comment 6 Kamil Dudka 2012-01-09 14:31:42 UTC
(In reply to comment #5)
> [root@z-sv-rheltst ~]# getent ahosts

I meant the output of 'getent ahost YOUR_MIRROR'.  Please attach also your /etc/nsswitch.conf and /etc/gai.conf.

Comment 7 Klaus Steinberger 2012-01-10 07:43:19 UTC
[root@z-sv-rheltst ~]# getent ahosts xmlrpc.rhn.redhat.com
209.132.183.44  STREAM xmlrpc.rhn.redhat.com
209.132.183.44  DGRAM  
209.132.183.44  RAW    
[root@z-sv-rheltst ~]# 

/etc/gai.conf is empty (I didn't change anything)

I will append /etc/nsswitch.conf

Comment 8 Klaus Steinberger 2012-01-10 07:43:50 UTC
Created attachment 551784 [details]
nsswitch.conf

Comment 9 Kamil Dudka 2012-01-10 10:56:07 UTC
Please export URLGRABBER_DEBUG=DEBUG and attach the output of both working and non-working cases as text files.

Comment 10 Klaus Steinberger 2012-01-10 11:33:35 UTC
Created attachment 551815 [details]
Not working case (options rotate in resolv.conf and only IPV6 DNS)

Not working case (options rotate in resolv.conf and only IPV6 DNS)

Comment 11 Klaus Steinberger 2012-01-10 11:34:25 UTC
Created attachment 551816 [details]
Working case (only IPV6 DNS, but no options line in resolv.conf)

Working case (only IPV6 DNS, but no options line in resolv.conf)

Comment 12 Kamil Dudka 2012-01-10 11:51:44 UTC
(In reply to comment #10)
> Created attachment 551815 [details]
> Not working case (options rotate in resolv.conf and only IPV6 DNS)

I cannot see any failure of libcurl in the non-working case.  Could this be narrowed down to a curl-based test-case?

Comment 13 Klaus Steinberger 2012-01-10 13:14:51 UTC
Its getting wierd more and more:

Some more testing on a Scientific linux host (as I have more control over yum):

2 or less IPV6 nameservers with options rotate fails, but curl works. 3 IPV6 nameservers are failing.

Here a failing yum session:

[root@z-sv-dns1 ~]# yum update
Repository sl-fastbugs is listed more than once in the configuration
internal                                                                                                                                                                                       | 1.1 kB     00:00     
internal/primary                                                                                                                                                                               |  12 kB     00:00     
internal                                                                                                                                                                                                        28/28
http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'sl.physik.lmu.de'"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: sl. Please verify its path and try again


Wget works:


[root@z-sv-dns1 ~]# wget http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml
--2012-01-10 14:09:26--  http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml
Auflösen des Hostnamen »sl.physik.lmu.de«.... 2001:4ca0:4102::1:34, 192.54.42.34
Verbindungsaufbau zu sl.physik.lmu.de|2001:4ca0:4102::1:34|:80... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 3307 (3,2K) [text/xml]
In »»repomd.xml.15«« speichern.

100%[============================================================================================================================================================================>] 3.307       --.-K/s   in 0,001s  

2012-01-10 14:09:26 (3,96 MB/s) - »»repomd.xml.15«« gespeichert [3307/3307]


Curl works too:


[root@z-sv-dns1 ~]# curl http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1311779765</revision>
  <data type="other_db">
    <location href="repodata/other.sqlite.bz2"/>
    <checksum type="sha256">45ff434da6c30d2f46500217e298a768b911227ecb2869d7db7862e7445e8fc9</checksum>
    <timestamp>1311779867.07</timestamp>
    <size>1892419</size>
    <open-size>11474944</open-size>
    <open-checksum type="sha256">a666818788ea9301c533d9adc882ea4fc693f2f8e60a9020579d026797b52492</open-checksum>
    <database_version>10</database_version>
  </data>
  <data type="other">
    <checksum type="sha256">a51438d73d4fc3c03c3cd8427240c4a1702b3ca835fbb5a7ca9a36ff27725864</checksum>
    <timestamp>1311779863</timestamp>
    <size>1227256</size>
    <open-size>12197914</open-size>
    <open-checksum type="sha256">a96571baeb21fd2c6735f2f2cde8deafafd807dd7e6a82961df2af97c24f3fd4</open-checksum>
    <location href="repodata/other.xml.gz"/>
  </data>
  <data type="filelists_db">
    <location href="repodata/filelists.sqlite.bz2"/>
    <checksum type="sha256">b2fad8e4d87c5430532c33ceed3be73f343b23886431df2db84c104123fc84db</checksum>
    <timestamp>1311779880.91</timestamp>
    <size>5201988</size>
    <open-size>32673792</open-size>
    <open-checksum type="sha256">1e9b00c12bfd18d4eccfe745a736c25a0ea5b8d83d3253e21096231f0ba6dcee</open-checksum>
    <database_version>10</database_version>
  </data>
  <data type="filelists">
    <checksum type="sha256">77e5d95c3a6f30f8c078598b36a301c0dbf61be7c7646bc660d238f8d10739f8</checksum>
    <timestamp>1311779863</timestamp>
    <size>4937567</size>
    <open-size>69945751</open-size>
    <open-checksum type="sha256">9e050ae0ae9f961bf6b45ef84df90dc48f43d30b9e79e14aa5ebad8397ba6713</open-checksum>
    <location href="repodata/filelists.xml.gz"/>
  </data>
  <data type="primary_db">
    <location href="repodata/primary.sqlite.bz2"/>
    <checksum type="sha256">65bd042497bbace157417c781d735a92d717aa91fba1f0f4fba006e7e661ca61</checksum>
    <timestamp>1311779889.48</timestamp>
    <size>4146868</size>
    <open-size>19913728</open-size>
    <open-checksum type="sha256">e3a4cd13113a7c6c8142768a59ef1a3a60668b298e0570678d6c7b3ab520d5bc</open-checksum>
    <database_version>10</database_version>
  </data>
  <data type="primary">
    <checksum type="sha256">4ef02f15480e16b9f40bdc385dcb8357efca586766a88d6f9f53b07ed09b19f1</checksum>
    <timestamp>1311779863</timestamp>
    <size>1938149</size>
    <open-size>16663128</open-size>
    <open-checksum type="sha256">3e60ecc392154ae88a0018ad091f60279d440ed9fadba2a8494ddcc202605d29</open-checksum>
    <location href="repodata/primary.xml.gz"/>
  </data>
  <data type="group_gz">
    <location href="repodata/comps-sl6-x86_64.xml.gz"/>
    <checksum type="sha256">f80a106c4baa2952da9453e20e185c097983d772ed26aa39e4dc07f8e558aeeb</checksum>
    <open-checksum type="sha256">1dd8f07850968f91b23df430db695fafccd3074042d0a6dd945b0e2f80adc99c</open-checksum>
    <timestamp>1311779889</timestamp>
  </data>
  <data type="group">
    <location href="repodata/comps-sl6-x86_64.xml"/>
    <checksum type="sha256">1dd8f07850968f91b23df430db695fafccd3074042d0a6dd945b0e2f80adc99c</checksum>
    <timestamp>1311779889</timestamp>
  </data>
</repomd>
[root@z-sv-dns1 ~]# 


I suspect now some very curious side effect inside the resolver libraries!

Comment 14 Kamil Dudka 2012-01-11 11:43:25 UTC
Is the failure reproducible independently on yum then?

Comment 15 Klaus Steinberger 2012-01-11 14:10:12 UTC
Yes, I can reproduce it with IBM Tivoli Storage Client:

[root@z-sv-rheltst adsm]# dsmc q filespace
ANS0102W Unable to open the message repository /opt/tivoli/tsm/client/ba/bin/DE_DE/dsmclientV3.cat. The American English repository will be used instead.
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 6, Release 3, Level 0.0  
  Client date/time: 11.01.2012 15:06:36
(c) Copyright by IBM Corporation and other(s) 1990, 2011. All Rights Reserved.

Node Name: Z-SV-RHELTST.PH
ANS1033E An invalid TCP/IP address was specified.

[root@z-sv-rheltst adsm]# 

In this case with only one IPV6 Nameserver and options rotate

and I was able to reproduce it with telnet client:

[root@z-sv-rheltst adsm]# telnet www.physik.uni-muenchen.de 80
telnet: www.physik.uni-muenchen.de: Name or service not known
www.physik.uni-muenchen.de: Host name lookup failure
[root@z-sv-rheltst adsm]# host www.physik.uni-muenchen.de
www.physik.uni-muenchen.de has address 192.54.42.2
www.physik.uni-muenchen.de has IPv6 address 2001:4ca0:4102::1:2
[root@z-sv-rheltst adsm]#

Comment 16 Kamil Dudka 2012-01-11 19:06:01 UTC
It is not a yum problem then.  I am switching the component to glibc although it can be a configuration/network problem.

Comment 17 Jeff Law 2012-03-01 19:14:54 UTC
Klaus,

The bad news, telnet explicitly only asks for IPV4 names because it simply doesn't support IPV6, so it's clearly not a good test.

The good news, I have a patch which *might* fix this problem, but I don't have a good test environment handy.  If I was to build you a version of glibc with the potential fix, could you test it and report the results here?

Comment 18 Klaus Steinberger 2012-03-01 21:16:11 UTC
Yes of course. Testing should be no problem

Comment 19 Klaus Steinberger 2012-03-04 09:24:23 UTC
Hi Jeff,

I tested the patch on a 64bit system, looks like it works.

Sincerly,
Klaus

Comment 20 Jeff Law 2012-03-04 17:50:43 UTC
Thanks for testing the patch.  I'm going to ack this from the dev side for 6.3; I'm not sure if the QA team has any bandwidth left though.

QA: If it helps, I think I can probably get a simpler testcase now that I know the root cause of this problem.

Comment 24 Jeff Law 2012-07-23 19:44:13 UTC
Same underlying issue as 841787, which is scheduled to be fixed in RHEL 6.4.

*** This bug has been marked as a duplicate of bug 841787 ***


Note You need to log in before you can comment on or make changes to this bug.