Bug 651592

+++ This bug was initially created as a clone of Bug #650255 +++

Description of problem:
If a Curl connection to an FTP server remains idle for a long time, the server may send a "421 Timeout". This should prompt Curl to drop the connection and re-connect for any subsequent transfers.

At the moment, Curl seems to ignore the 421 and gets confused when later trying to send request to the previous command channel.


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. If possible, adjust timeout for FTP server. E.g vsftpd has an idle_session_timeout. Make this small e.g. 60 seconds.

2. Start a Fedora 14 installation via an FTP server. Wait at the package selection page for longer than the timeout.

3.
  
Actual results:
Installer fails to download filelists.xml.gz due to timeout problem.

Expected results:
Curl re-connects after server error.


Additional info:

--- Additional comment from kdudka on 2010-11-09 14:20:26 CET ---

I was playing with that and it seems to work in most cases.  I set up a testing FTP server (proftpd) with idle timeout set to 10:

# grep Idle /etc/proftpd.conf
TimeoutIdle 10

Then I prepared a testing client to repeat the issue.  It performs two FTP downloads using a specified delay between them.  If the delay is 8 or 12, it works fine.  If the delay is 10, it does not.  Am I observing the same issue as you with Anaconda?

--- Additional comment from kdudka on 2010-11-09 14:22:41 CET ---

Created attachment 459098 [details]
a testing client

--- Additional comment from kdudka on 2010-11-09 14:30:22 CET ---

Created attachment 459099 [details]
output of the testing client

taken as;

$ curl -sJO 'https://bugzilla.redhat.com/attachment.cgi?id=459098'
$ sh bz650255.c 2>&1 | tee bz650255.log

--- Additional comment from kdudka on 2010-11-09 14:32:27 CET ---

Comment on attachment 459098 [details]
a testing client

Oops, one typo in there, but it still works:

s/exho/echo/

--- Additional comment from kdudka on 2010-11-09 14:42:58 CET ---

Comment on attachment 459099 [details]
output of the testing client

The following part of the log is interesting for us:

> * Re-using existing connection! (#0) with host localhost
> * Connected to localhost (127.0.0.1) port 21 (#0)
> * Request has same path as previous transfer
> > EPSV
> * Connect data stream passively
> < 421 Idle timeout (10 seconds): closing control connection
> * disabling EPSV usage
> > PASV
> * Recv failure: Connection reset by peer
> * Closing connection #0
> * Failure when receiving data from the peer
> + exit 56

--- Additional comment from number.cruncher on 2010-11-09 14:48:04 CET ---

Created attachment 459106 [details]
Client output connecting to VSFTPD server

Output when re-connecting before timeout.

--- Additional comment from kdudka on 2010-11-09 15:07:20 CET ---

Could you please try to adjust the timeout, such that it reflects your FTP server's timeout?

diff --git a/bz650255.c b/bz650255.c
index ca11705..89de371 100644
--- a/bz650255.c
+++ b/bz650255.c
@@ -1,12 +1,12 @@
 #if 0
 gcc -pedantic -Wall -Wextra -O0 -g bz650255.c -lcurl || exit $?
-./a.out 8  || exit $?
-./a.out 12 || exit $?
+./a.out 58 || exit $?
+./a.out 62 || exit $?
 echo
 echo === still OK ===
-exho
+echo
 set -x
-./a.out 10
+./a.out 60
 exit $?
 #endif

--- Additional comment from kdudka on 2010-11-09 17:11:06 CET ---

Created attachment 459170 [details]
proposed fix

--- Additional comment from kdudka on 2010-11-09 17:19:41 CET ---

The attached fix at least helps to return a more appropriate error code in case of the timeout on server's side.  I am not sure if libcurl itself should try to reconnect in that case...

Daniel, what do you think about the issue?

Is the behavior somehow specified in libcurl's API?

--- Additional comment from number.cruncher on 2010-11-09 17:38:04 CET ---

I am having a hard time reproducing this with your client code. Perhaps the anaconda code is taking a different code path? Can't fault the above fix, though!

--- Additional comment from kdudka on 2010-11-09 17:46:42 CET ---

Well, let's fix the issue with proftpd and my reproducer for now.  Then we can retest with Anaconda.

Based on your description, I suspect that Anaconda holds a libcurl handle across all the user interaction during the package selection.  To be honest, I don't know if we are supposed to handle such cases transparently within libcurl and silently reconnect on server's idle timeout ... but the response in case of my reproducer looks wrong to me.

--- Additional comment from daniel on 2010-11-09 18:01:00 CET ---

(In reply to comment #9)
> The attached fix at least helps to return a more appropriate error code in case
> of the timeout on server's side.  I am not sure if libcurl itself should try to
> reconnect in that case...
> 
> Daniel, what do you think about the issue?
> 
> Is the behavior somehow specified in libcurl's API?

I think that's a fair approach. 421 is documented in RFC959 to be exactly "we're closing down the control connection" and it can come at any time so it makes sense to put the logic in the common response reader function.

I don't think libcurl needs to/should reconnect by itself.

Please go ahead and commit this upstream as well!

--- Additional comment from kdudka on 2010-11-09 18:24:51 CET ---

Daniel, thanks for looking at the issue.  I've just pushed the patch upstream:

https://github.com/bagder/curl/commit/12b2412

Anaconda developers have IMO two choices:

1) avoid holding the libcurl handle during packages selection

2) pick up the error code and reconnect on their own responsibility eventually

--- Additional comment from kdudka on 2010-11-09 18:33:16 CET ---

fixed in curl-7.21.2-4.fc15

--- Additional comment from clumens on 2010-11-09 19:12:13 CET ---

Such a change would need to be made somewhere farther down in the pile, like either in yum or python-urlgrabber.  anaconda does not directly work with libcurl in the package downloading portion, aside from resetting the name server cache stuff.