Bug 853432 - Yum with parallel download cannot download packages from repo that allow only single connection
Yum with parallel download cannot download packages from repo that allow only...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: python-urlgrabber (Show other bugs)
7.0
All Linux
medium Severity medium
: rc
: ---
Assigned To: James Antill
Patrik Kis
: Regression
Depends On:
Blocks: 851152
  Show dependency treegraph
 
Reported: 2012-08-31 09:41 EDT by Patrik Kis
Modified: 2017-06-27 02:39 EDT (History)
5 users (show)

See Also:
Fixed In Version: python-urlgrabber-3.10-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-06-13 06:49:11 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
urlgrabber log (30.86 KB, text/plain)
2013-10-04 08:07 EDT, Patrik Kis
no flags Details
urlgrabber log from booth release 30 and 31 (4.53 KB, application/x-gzip)
2013-10-07 10:50 EDT, Patrik Kis
no flags Details
Logs from test with python-urlgrabber-3.9.1-32.el7 (6.81 KB, application/x-gzip)
2013-10-08 08:45 EDT, Patrik Kis
no flags Details
Logs from test with python-urlgrabber-3.9.1-33.el7 (6.62 KB, application/x-gzip)
2013-10-08 10:07 EDT, Patrik Kis
no flags Details
Logs from test with python-urlgrabber-3.10-0.el7 (7.58 KB, application/x-gzip)
2013-10-17 11:04 EDT, Patrik Kis
no flags Details
Logs from RHEL-6 (10.12 KB, application/x-gzip)
2013-10-18 05:11 EDT, Patrik Kis
no flags Details
Logs from test with python-urlgrabber-3.10-1.el7 (13.58 KB, application/x-gzip)
2013-10-22 04:06 EDT, Patrik Kis
no flags Details

  None (edit)
Description Patrik Kis 2012-08-31 09:41:25 EDT
Description of problem:
The new yum in RHEL-7, which is capable of parallel package download, fail to download and install packages from a repository which allows only one connection. 

Version-Release number of selected component (if applicable):
yum-3.4.3-32.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. Set up a ftp repo with vsftpd
# cat /etc/yum.repos.d/ftp_test.repo 
[ftp_test]
name=ftp_test
baseurl=ftp://127.0.0.1/ftp_test/
enabled=1
gpgcheck=0
# ls -l /var/ftp/ftp_test/
total 5604
-rw-r--r--. 1 root root 716328 Aug 31 15:25 parayum1-1.0-1.noarch.rpm
-rw-r--r--. 1 root root 716324 Aug 31 15:25 parayum2-1.0-1.noarch.rpm
-rw-r--r--. 1 root root 716324 Aug 31 15:25 parayum3-1.0-1.noarch.rpm
-rw-r--r--. 1 root root 716328 Aug 31 15:25 parayum4-1.0-1.noarch.rpm
-rw-r--r--. 1 root root 716328 Aug 31 15:25 parayum5-1.0-1.noarch.rpm
-rw-r--r--. 1 root root 716328 Aug 31 15:25 parayum6-1.0-1.noarch.rpm
-rw-r--r--. 1 root root 716328 Aug 31 15:25 parayum7-1.0-1.noarch.rpm
-rw-r--r--. 1 root root 716324 Aug 31 15:25 parayum8-1.0-1.noarch.rpm
drwxr-xr-x. 2 root root   4096 Aug 31 15:25 repodata

2. Add to /etc/vsftpd/vsftpd.conf
max_clients=1

3. Start the vsftpd server

4. Install the packages from this repo
# yum -y install parayum1.noarch parayum2.noarch
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Resolving Dependencies
--> Running transaction check
---> Package parayum1.noarch 0:1.0-1 will be installed
---> Package parayum2.noarch 0:1.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================
 Package                      Arch                       Version                      Repository                    Size
=========================================================================================================================
Installing:
 parayum1                     noarch                     1.0-1                        ftp_test                     700 k
 parayum2                     noarch                     1.0-1                        ftp_test                     700 k

Transaction Summary
=========================================================================================================================
Install  2 Packages

Total download size: 1.4 M
Installed size: 1.4 M
Downloading Packages:
(1/2): parayum1-1.0-1.noarch.rpm                                                                  | 700 kB     00:00     
parayum2-1.0-1.noarch.rpm      FAILED Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.


Error Downloading Packages:
  parayum2-1.0-1.noarch: Downloaded package parayum2-1.0-1.noarch, from ftp_test, but it was invalid.
  parayum2-1.0-1.noarch: [Errno 256] No more mirrors to try.

5. Just to confirm that the repo is ok:
# yum -y install parayum1.noarch 
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Resolving Dependencies
--> Running transaction check
---> Package parayum1.noarch 0:1.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================
 Package                      Arch                       Version                      Repository                    Size
=========================================================================================================================
Installing:
 parayum1                     noarch                     1.0-1                        ftp_test                     700 k

Transaction Summary
=========================================================================================================================
Install  1 Package

Total size: 700 k
Installed size: 700 k
Downloading Packages:
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : parayum1-1.0-1.noarch                                                                                 1/1 
  Verifying  : parayum1-1.0-1.noarch                                                                                 1/1 

Installed:
  parayum1.noarch 0:1.0-1                                                                                                

Complete!


  
Actual results:
the packages are not installed

Expected results:
install the a packages regardless of number of allowed connections
Comment 4 Zbysek MRAZ 2013-05-09 08:58:27 EDT
This bug is still present in recent tree of RHEL7:

RHEL-7.0-20130502.0 Server
yum-3.4.3-86.el7
Comment 5 Zdeněk Pavlas 2013-05-09 09:50:58 EDT
This repository configuration does not use metalink.xml that specifies the per-mirror connection limit.  As a result, Yum uses the urlgrabber default (that's up to 2 connections to the same mirror).  Also, max_connections is not set in yum.conf so the effective total connection limit is 5 (again, the urlgrabber default).

IMO, Yum behaves as specified and expected.  Does adding max_connections=1 to yum.conf solve the problem?  Is that an acceptable solution?

To handle such cases without user intervention, maybe we should handle server timeouts the same way as HTTP 503 errors, so Yum will retry the same mirror up to retry times. (see the related BZ 765598).
Comment 6 Patrik Kis 2013-05-10 05:15:16 EDT
(In reply to comment #5)
> This repository configuration does not use metalink.xml that specifies the
> per-mirror connection limit.  As a result, Yum uses the urlgrabber default
> (that's up to 2 connections to the same mirror).  Also, max_connections is
> not set in yum.conf so the effective total connection limit is 5 (again, the
> urlgrabber default).
> 
> IMO, Yum behaves as specified and expected.  Does adding max_connections=1
> to yum.conf solve the problem?  Is that an acceptable solution?

Not really. This test is about backward compatibility and regression. Yum prior parallel download would work with this specific repo with default setting. The new yum should work out of the box as well without configuration changes.

> 
> To handle such cases without user intervention, maybe we should handle
> server timeouts the same way as HTTP 503 errors, so Yum will retry the same
> mirror up to retry times. (see the related BZ 765598).

Timeout sounds good, the question is what happen when the connection times out completely, i.e. 3x timeout. For example, if the timeout is 5 second and download time for one package takes 30 seconds, yum starts to download the 1st package but the 2bd connection times out completely after 15 seconds (3x5seconds). After this point I can imagine two scenarios:
1/ Installation fail; IMHO we should avoid this.
2/ Kind of fallback to single connection download; i.e. yum restart to 2nd package download after the 1st package download finished.
Comment 7 Zdeněk Pavlas 2013-05-10 06:35:59 EDT
Other possibility is to change the default same-mirror connection limit to 1 in urlgrabber.  Realistically, is ftpd with max_clients=1 something we must support ootb?  The max_clients option is very likely a global limit.  Such server won't work with old Yum either, if there's another client active.

> For example, if the timeout is 5 second and download time for one package takes 30 seconds, yum starts to download the 1st package but the 2bd connection times out completely after 15 seconds (3x5seconds).

The default timeout is 30 seconds, and the default retry limit is 10.  It should work in most cases, IMO.
Comment 8 Patrik Kis 2013-05-10 08:10:56 EDT
(In reply to comment #7)
> Other possibility is to change the default same-mirror connection limit to 1
> in urlgrabber.  Realistically, is ftpd with max_clients=1 something we must
> support ootb?  The max_clients option is very likely a global limit.  Such
> server won't work with old Yum either, if there's another client active.

The ftp setting max_clients=1 is used only for test purposes. In production it probably wouldn't be used this way (although it is still a valid configuration), but what could be set up is a connection limit per host, e.g. on firewall or in ftp/http server tself. The result would be the same: only one connection allowed per client.

> 
> > For example, if the timeout is 5 second and download time for one package takes 30 seconds, yum starts to download the 1st package but the 2bd connection times out completely after 15 seconds (3x5seconds).
> 
> The default timeout is 30 seconds, and the default retry limit is 10.  It
> should work in most cases, IMO.

Ok that's about 5 minutes; pretty much. I think then the timeout/retry approach could work for the most real word cases.
Comment 9 Patrik Kis 2013-06-17 09:27:16 EDT
Verification: failed
When I try to install more packages from a repository that allows only one ftp connection the installation fails.
yum repolist all succeed, but a lot of ugly error messages are displayed for every retry. Basically error all messages says the same, so maybe one warning would be enough.

yum-3.4.3-91.el7.noarch
python-urlgrabber-3.9.1-28.el7.noarch

The same reproducer as from C1, i.e. a simple ftp repo, default vsftpd.conf just three lines added:
anon_max_rate=100000
trans_chunk_size=4096
max_clients=1


# yum repolist all
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
beaker-client                                                                   | 1.5 kB  00:00:00     
ftp_test                                                                        | 2.9 kB  00:00:00     
qa-tools                                                                        | 1.2 kB  00:00:00     
rhel7                                                                           | 3.8 kB  00:00:00     
rhel7-debug                                                                     | 3.0 kB  00:00:00     
rhel7-optional                                                                  | 3.8 kB  00:00:00     
rhel7-optional-debug                                                            | 3.0 kB  00:00:00     
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
(1/7): qa-tools/primary                                                         |  31 kB  00:00:00     
(2/7): rhel7-debug/primary_db                                                   | 582 kB  00:00:00     
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
(3/7): rhel7/primary_db                                                         | 2.8 MB  00:00:00     
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
(4/7): beaker-client/primary                                                    | 5.3 kB  00:00:00     
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
(5/7): rhel7-optional/primary_db                                                | 3.3 MB  00:00:01     
(6/7): rhel7-optional-debug/primary_db                                          |  56 kB  00:00:03     
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/b67b877ffd88e90f7e4f72d1930cd65ecaff45c24ce4f53cde99c336ebcc1868-primary.sqlite.bz2: (28, '')
Trying other mirror.
beaker-client                                                                                    28/28
ftp_test/primary_db                                                             | 1.9 kB  00:00:00     
qa-tools                                                                                       243/243
repo id                              repo name                                           status
beaker-client                        Beaker Client - Fedora7.0                           enabled:    28
beaker-client-testing                Beaker Client - Fedora7.0 - Testing                 disabled
ftp_test                             ftp_test                                            enabled:     4
qa-tools                             QA Tools                                            enabled:   243
rhel7                                rhel7                                               enabled: 4,327
rhel7-debug                          rhel7-debug                                         enabled: 1,894
rhel7-optional                       rhel7-optional                                      enabled: 7,977
rhel7-optional-debug                 rhel7-optional-debug                                enabled:   226
repolist: 14,699
# 
#
#
# yum install -y parayum1 parayum2 parayum3 parayum4
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Resolving Dependencies
--> Running transaction check
---> Package parayum1.noarch 0:1.0-1 will be installed
---> Package parayum2.noarch 0:1.0-1 will be installed
---> Package parayum3.noarch 0:1.0-1 will be installed
---> Package parayum4.noarch 0:1.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================
 Package                  Arch                   Version                Repository                Size
=======================================================================================================
Installing:
 parayum1                 noarch                 1.0-1                  ftp_test                 706 k
 parayum2                 noarch                 1.0-1                  ftp_test                 706 k
 parayum3                 noarch                 1.0-1                  ftp_test                 706 k
 parayum4                 noarch                 1.0-1                  ftp_test                 706 k

Transaction Summary
=======================================================================================================
Install  4 Packages

Total download size: 2.8 M
Installed size: 2.8 M
Downloading packages:
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum2-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum3-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum3-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
parayum4-1.0-1.noarch.rpm      FAILED                                          
ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/parayum4-1.0-1.noarch.rpm: (28, '')
Trying other mirror.
(1/4): parayum1-1.0-1.noarch.rpm                                                | 706 kB  00:00:07     


Error downloading packages:
  parayum4-1.0-1.noarch: [Errno 256] No more mirrors to try.
  parayum3-1.0-1.noarch: [Errno 256] No more mirrors to try.
  parayum2-1.0-1.noarch: [Errno 256] No more mirrors to try.


Tcpdump showed that yum while downloading the 1st package, yum tries to establish a new ftp connection to port 21 but that is rejected constantly by the ftp server.
After 20 retry or so, yum gives it up and after the 1st package download finish, it fails with installation.
It looks like the retry/timeout approach does not work when the connection is rejected immediately because the retry limit is used quickly. So I think other approach is necessary. Please note that the same test works fine on RHEL-6, with yum prior parallel download approach. IMHO, yum should simple retry again after the download of the 1st package finished and the connection is closed with ftp server.
And I think I should extend the test to http protocol too.
Comment 10 Patrik Kis 2013-06-17 10:04:56 EDT
When the connection limitation is done by iptables and not by vsftpd, like:

iptables -I INPUT 1 -p tcp --syn --dport 21 -m connlimit --connlimit-above 1 -j REJECT --reject-with tcp-reset

the fails again. On RHEL-6 it works.
Comment 11 Zdeněk Pavlas 2013-06-17 10:25:49 EDT
> the connection is rejected immediately
...
> ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: [Errno 12] Timeout on
> ftp://127.0.0.1/ftp_test/parayum2-1.0-1.noarch.rpm: (28, '')
> Trying other mirror.

Weird, curl raised error=28 timeout, so it should be 30 seconds or so.  Probably the FTP server responds with something like "too many connections, go away", but curl reports it (incorrectly) as timeout.

> IMHO, yum should simple retry again after the download of the 1st package finished and the connection is closed with ftp server.

Yum can't know that the pending download (parayum1) IS CAUSING other downloads to fail.  The downloader is aware that these are using the same server, but since the connection limit is 2, they're tried in parallel.  Seems that the only possible solutions are:

a) Add a retry-after-timeout queue.  Some errors (HTTP 530, FTP timeout) are not retried immediately, but after a configured delay.  We could also parse the suggested delay from HTTP 530 responses, and handle it "the right way"..

b) Adjust the host connection limit on timeouts.. That way, we'd change the connection limit from 2 to 1 after first timeout, and then block until parayum1  finishes.  This is much simpler and probably more robust.

I'd probably go with implementing b).
Comment 12 Patrik Kis 2013-06-17 10:49:42 EDT
(In reply to Zdeněk Pavlas from comment #11)
...
> 
> a) Add a retry-after-timeout queue.  Some errors (HTTP 530, FTP timeout) are
> not retried immediately, but after a configured delay.  We could also parse
> the suggested delay from HTTP 530 responses, and handle it "the right way"..
> 
> b) Adjust the host connection limit on timeouts.. That way, we'd change the
> connection limit from 2 to 1 after first timeout, and then block until
> parayum1  finishes.  This is much simpler and probably more robust.
> 
> I'd probably go with implementing b).

Yes, b) quite makes sense to me. If I understood correctly that would also prevent the spam of error messages.
Comment 13 Patrik Kis 2013-08-26 09:23:25 EDT
The verification failed again. I see no progress here with the new version of urlgrabber.

python-urlgrabber-3.9.1-29.el7
yum-3.4.3-101.el7

# yum repolist all
Loaded plugins: auto-update-debuginfo, product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
beaker-client                                                                                 | 1.5 kB  00:00:00     
ftp_test                                                                                      | 2.9 kB  00:00:00     
qa-tools                                                                                      | 1.2 kB  00:00:00     
rhel7                                                                                         | 3.8 kB  00:00:00     
rhel7-debug                                                                                   | 3.0 kB  00:00:00     
rhel7-optional                                                                                | 3.8 kB  00:00:00     
rhel7-optional-debug                                                                          | 3.0 kB  00:00:00     
http://127.0.0.1/testrepo/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to 127.0.0.1:80; Connection refused"
Trying other mirror.
beaker-client/primary                                                                         | 5.1 kB  00:00:00     
beaker-client                                                                                                  28/28
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db            FAILED                                          
ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: [Errno 12] Timeout on ftp://127.0.0.1/ftp_test/repodata/24a0745eebece8128116e97564c3e7efbf544f938592efb615267ad0aa54ed70-primary.sqlite.bz2: (28, '')
Trying other mirror.
ftp_test/primary_db                                                                           | 1.9 kB  00:00:00     
qa-tools/primary                                                                              |  32 kB  00:00:00     
qa-tools                                                                                                     263/263
rhel7/primary_db                                                                              | 3.4 MB  00:00:00     
rhel7-debug/primary_db                                                                        | 565 kB  00:00:00     
rhel7-optional/primary_db                                                                     | 1.7 MB  00:00:00     
rhel7-optional-debug/primary_db                                                               |  56 kB  00:00:00     
http://127.0.0.1/testrepo/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to 127.0.0.1:80; Connection refused"
Trying other mirror.
repo id                                     repo name                                                  status
beaker-client                               Beaker Client - Fedora7.0                                  enabled:    28
beaker-client-testing                       Beaker Client - Fedora7.0 - Testing                        disabled
ftp_test                                    ftp_test                                                   enabled:     4
qa-tools                                    QA Tools                                                   enabled:   263
rhel7                                       rhel7                                                      enabled: 4,390
rhel7-debug                                 rhel7-debug                                                enabled: 1,890
rhel7-optional                              rhel7-optional                                             enabled: 3,886
rhel7-optional-debug                        rhel7-optional-debug                                       enabled:   230
testrepo                                    testrepo                                                   enabled:     0
repolist: 10,691
Comment 14 Zdeněk Pavlas 2013-08-26 09:49:11 EDT
> ftp_test/primary_db            FAILED

But now it's NOT failing on *.rpm download, it fails on *-primary.sqlite.bz2 download instead!  I think the problem is that the FTP speed limit is too low, probably lower than the default minrate=1000.  Please try using FTP speed limit larger than Yum's minrate (or set a lower minrate in yum.conf).

(In older urlgrabber the default minrate was 1, now it's 1000), see BZ 964298.
Comment 15 Zdeněk Pavlas 2013-08-26 09:51:01 EDT
Sorry, should be BZ 860181.
Comment 16 Karel Srot 2013-08-29 08:04:55 EDT
(In reply to Zdeněk Pavlas from comment #14)
> > ftp_test/primary_db            FAILED
> 
> But now it's NOT failing on *.rpm download, it fails on *-primary.sqlite.bz2
> download instead!  I think the problem is that the FTP speed limit is too
> low, probably lower than the default minrate=1000.  Please try using FTP
> speed limit larger than Yum's minrate (or set a lower minrate in yum.conf).
> 
> (In older urlgrabber the default minrate was 1, now it's 1000), see BZ
> 964298.

Quick test shows that it didn't help. Tested with yum-3.4.3-101.el7.noarch.
Comment 17 Zdeněk Pavlas 2013-08-30 07:46:33 EDT
> ftp_test/primary_db | 1.9 kB  00:00:00

Ok, so the metadata request for "ftp_test/primary_db" finally succeeds:

> http://127.0.0.1/testrepo/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to 127.0.0.1:80; Connection refused"

Why are your testing ftp_test and testrepo in the same run?  Also, I suppose this test should primarily test install of multiple packages, not the "yum repolist" command.
Comment 19 Zdeněk Pavlas 2013-08-30 08:38:33 EDT
> http://127.0.0.1/testrepo/parayum1-1.0-1.noarch.rpm: [Errno 14] curl#7 - "Failed connect to 127.0.0.1:80; Connection refused"

Last time, the setup used FTP..  Why have you changed testrepo.baseurl to HTTP?  You should probably start httpd first. "nejsou penize, neni laska"!
Comment 20 Patrik Kis 2013-09-02 06:23:29 EDT
(In reply to Zdeněk Pavlas from comment #19)
> > http://127.0.0.1/testrepo/parayum1-1.0-1.noarch.rpm: [Errno 14] curl#7 - "Failed connect to 127.0.0.1:80; Connection refused"
> 
> Last time, the setup used FTP..  Why have you changed testrepo.baseurl to
> HTTP?  You should probably start httpd first. "nejsou penize, neni laska"!

There are 3 test in this automated test. In all of them a test repository is set up with a few packages, then "yum repolist all" is issued and finally the packages are installed then erased.
The difference between the tests are in how the repositories are accessible.

1/ ftp repository; only one ftp connection is allowed by the server:
        # echo 'anon_max_rate=100000' >> /etc/vsftpd/vsftpd.conf
        # echo 'trans_chunk_size=4096' >> /etc/vsftpd/vsftpd.conf
        # echo 'max_clients=1' >> /etc/vsftpd/vsftpd.conf

2/ ftp repository; used the default vsftdp.conf but the connection is limited by iptables
        # iptables -I INPUT 1 -p tcp --syn --dport 21 -m connlimit --connlimit-above 1 -j REJECT --reject-with tcp-reset

3/ the same as above (iptables limitation) but with httpd
        # iptables -I INPUT 1 -p tcp --syn --dport 80 -m connlimit --connlimit-above 1 -j REJECT --reject-with tcp-reset

Actually all 3 test cases are failing on RHEL-7; just for comparison, the very same test is passing on RHEL-6.4.
Originally there was only one test (FTP and not iptables) but then I extended it to cover more use cases.
Comment 21 Patrik Kis 2013-09-02 06:27:53 EDT
(In reply to Zdeněk Pavlas from comment #14)
> > ftp_test/primary_db            FAILED
> 
> But now it's NOT failing on *.rpm download, it fails on *-primary.sqlite.bz2
> download instead!  I think the problem is that the FTP speed limit is too
> low, probably lower than the default minrate=1000.  Please try using FTP
> speed limit larger than Yum's minrate (or set a lower minrate in yum.conf).
> 
> (In older urlgrabber the default minrate was 1, now it's 1000), see BZ
> 964298.

And I tried to play with booth minrate in yum.conf and also ftp speed too, but I observed no difference. yum repolist simply drops all those errors and the subsequent install fails.
Comment 22 Zdeněk Pavlas 2013-09-02 09:59:00 EDT
Thanks for explaining this. So now instead of stalling the FTP connection (and timing the client out) you reset it with iptables. This explains the 2nd error (Connection refused). I'll have to fall back to single-connection mode on refused connections, too.
Comment 23 Zdeněk Pavlas 2013-10-03 09:21:27 EDT
I couldn't push this fix to rhel, due to qa_ack=? .. Could you try with any of: python-urlgrabber-3.9.1-32.fc{19,20,21}? (you should be able to install it in rhel7)

These revert to single-connection mode after refused connections, too.  I hope this fixes this bug for good.  If not, please enable logging (export URLGRABBER_DEBUG=INFO,/tmp/urlgrabber.log), and attach the log, please.
Comment 26 Zdeněk Pavlas 2013-10-04 04:13:40 EDT
Could you turn on urlgrabber debugging, and attach the log, please?
(export URLGRABBER_DEBUG=INFO,/tmp/urlgrabber.log), re comment 23
Comment 27 Patrik Kis 2013-10-04 08:07:05 EDT
Created attachment 807602 [details]
urlgrabber log
Comment 28 Patrik Kis 2013-10-04 08:07:52 EDT
(In reply to Zdeněk Pavlas from comment #26)
> Could you turn on urlgrabber debugging, and attach the log, please?
> (export URLGRABBER_DEBUG=INFO,/tmp/urlgrabber.log), re comment 23

Sorry, I missed that comment. Here the debug log comes: https://bugzilla.redhat.com/attachment.cgi?id=807602
Comment 29 Zdeněk Pavlas 2013-10-07 08:23:52 EDT
Please, can you double check that you use python-urlgrabber-3.9.1-31.el7 ???
The per-host connection limit processing is NOT shown:

attached:

2013-10-04 08:04:53,321 max_connections: 0/5
2013-10-04 08:04:53,336 attempt 1/10: http://127.0.0.1/testrepo/repodata/07bc0819a0001189b63d19c2561e352cc57c07f84744ea7fdd7b4f8d84b216d2-primary.sqlite.bz2

expected:

2013-10-07 14:04:41,778 max_connections: 0/5
2013-10-07 14:04:41,779 max_connections(http://127.0.0.1/testrepo/): 0/1
2013-10-07 14:04:41,783 attempt 1/10: http://127.0.0.1/testrepo/...

There's no code path that could skip the per-host max_mirrors debug output.
Comment 30 Patrik Kis 2013-10-07 10:50:33 EDT
Created attachment 808862 [details]
urlgrabber log from booth release 30 and 31
Comment 31 Patrik Kis 2013-10-07 11:10:44 EDT
(In reply to Patrik Kis from comment #30)
> Created attachment 808862 [details]
> urlgrabber log from booth release 30 and 31

it seems that the required log item is missing from log of python-urlgrabber-3.9.1-31.el7; see the latest attachment
Comment 32 Zdeněk Pavlas 2013-10-08 06:38:53 EDT
Please try with python-urlgrabber-3.9.1-32.el7, the single-connection mode was not activated when max_connections was not set explicitly and defaulted to 2 connections.  It still does not explain the missing per-host limit debug output, but let's see..
Comment 33 Patrik Kis 2013-10-08 08:45:38 EDT
Created attachment 809251 [details]
Logs from test with python-urlgrabber-3.9.1-32.el7
Comment 34 Patrik Kis 2013-10-08 08:46:36 EDT
It does not look much better; see the attachment.
Comment 35 Zdeněk Pavlas 2013-10-08 09:27:03 EDT
Please try with python-urlgrabber-3.9.1-33.el7
Comment 36 Patrik Kis 2013-10-08 10:07:42 EDT
Created attachment 809321 [details]
Logs from test with python-urlgrabber-3.9.1-33.el7
Comment 38 Patrik Kis 2013-10-17 11:02:58 EDT
The test failed again. I'll attach the logs, but if you'd like I can provide you with a test machine with the test so it can be re-run while fixing the component.
Comment 39 Patrik Kis 2013-10-17 11:04:11 EDT
Created attachment 813384 [details]
Logs from test with python-urlgrabber-3.10-0.el7
Comment 40 Zdeněk Pavlas 2013-10-18 04:40:32 EDT
Please, could you provide also logs from the same test but with rhel6 package?
Comment 41 Patrik Kis 2013-10-18 05:11:12 EDT
Created attachment 813657 [details]
Logs from RHEL-6
Comment 42 Zdeněk Pavlas 2013-10-21 11:08:34 EDT
Finally found the root problem.. Yum downloads repomd.xml with the legacy single-connection downloader, and *then* tries to download metadata (or packages) with the multi-downloader.  But the repomd.xml connection is keep-alive'd, so no matter how hard the multi-downloader tries to use only one connection, it's still rejected by the server.  Added a flush of the single-downloader connection.
Comment 43 Patrik Kis 2013-10-22 04:04:55 EDT
If the the directory /var/cache/yum is removed and yum repolist all is run prior the tests the 1st part of the test pass (when the connection is limited by max_clients=1 in vsftpd.conf), but the 2nd and 3rd test still fails (ftp and http repo where the connection is limited by a rule in iptables).
With the second run the test fails completely (if cached data were not cleared).
Comment 44 Patrik Kis 2013-10-22 04:06:01 EDT
Created attachment 814883 [details]
Logs from test with python-urlgrabber-3.10-1.el7
Comment 45 Zdeněk Pavlas 2013-10-24 10:25:13 EDT
:: [   LOG    ] :: Test yum with repo that supports only one connection: ftp connection limited by firewall
:: [   PASS   ] :: Running 'iptables -I INPUT 1 -p tcp --syn --dport 21 -m connlimit --connlimit-above 1 -j REJECT --reject-with tcp-reset' (Expected 0, got 0)

I believe this test is incorrect- the firewall probably includes close-wait connections in the total count, too.  So once Yum uses two connections, for a minute or so every new connection is being reset, even if all others were closed.
The test worked on EL6, becuase Yum used only one connection during the whole session.

There's other problem: Yum retries the same mirror on timeouts and 503 errors, but refused connection (with tcp reset) is considered permanent.  Even when I modify Yum to retry on such errors, the test does not pass.  But it's that the firewall rejects retries EVEN AFTER the first download has already finished.
Comment 46 Patrik Kis 2013-10-25 06:15:59 EDT
(In reply to Zdeněk Pavlas from comment #45)
> :: [   LOG    ] :: Test yum with repo that supports only one connection: ftp
> connection limited by firewall
> :: [   PASS   ] :: Running 'iptables -I INPUT 1 -p tcp --syn --dport 21 -m
> connlimit --connlimit-above 1 -j REJECT --reject-with tcp-reset' (Expected
> 0, got 0)
> 
> I believe this test is incorrect- the firewall probably includes close-wait
> connections in the total count, too.  So once Yum uses two connections, for
> a minute or so every new connection is being reset, even if all others were
> closed.
> The test worked on EL6, becuase Yum used only one connection during the
> whole session.
> 
> There's other problem: Yum retries the same mirror on timeouts and 503
> errors, but refused connection (with tcp reset) is considered permanent. 
> Even when I modify Yum to retry on such errors, the test does not pass.  But
> it's that the firewall rejects retries EVEN AFTER the first download has
> already finished.

True; I fixed the firewall rules.
Comment 47 Zdeněk Pavlas 2013-10-25 08:17:31 EDT
Please re-test with fixed firewall rules, and with yum-3.4.3-104.el7 that retries refused connections.
Comment 49 Ludek Smid 2014-06-13 06:49:11 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.