Bug 336151

Summary: RHN Proxy Problems
Product: Red Hat Satellite 5 Reporter: Issue Tracker <tao>
Component: ServerAssignee: Miroslav Suchý <msuchy>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: high Docs Contact:
Priority: high    
Version: 510CC: cperry, msuchy, tao
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: sat510 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-04-02 20:31:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 248630, 317791    

Description Issue Tracker 2007-10-17 13:33:57 UTC
Escalated to Bugzilla from IssueTracker

Comment 1 Issue Tracker 2007-10-17 13:33:59 UTC
Description of problem:  Paul,  I'm having problems with a server provision from one of my RHN Proxies.  The server was able to get 75% of the way through the installation and then started receiving 302 errors from the proxy for the "compat-db" package... I'm not sure that it has anything to do with that package... maybe something going on with the proxy... I've attached a proxy debug... 

Can you take a look at this please?

Andy
This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 2 Issue Tracker 2007-10-17 13:34:00 UTC
Customer noted to me on the phone that he was also getting errors about
"too many redirects" when he tried to manually download the package
through the proxy using wget and whatever URL it was choking on.

-- 
paul novarese


This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 3 Issue Tracker 2007-10-17 13:34:00 UTC
File uploaded: brspoa24l-sysreport.tar.bz2

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237
it_file 104321

Comment 4 Issue Tracker 2007-10-17 13:34:02 UTC
File uploaded: utwwdc12-sysreport.tar.bz2

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237
it_file 104322

Comment 5 Issue Tracker 2007-10-17 13:34:03 UTC
Paul,

brspoa24l - RHN proxy
utwwdc12  - RHSS

Andy


This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 6 Issue Tracker 2007-10-17 13:34:03 UTC
File uploaded: satellite-debug.tar.bz2

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237
it_file 104329

Comment 7 Issue Tracker 2007-10-17 13:34:04 UTC
All ITs
State the problem

   1. Provide time and date of problem

intermittent

   2. Provide clear and concise problem description as it is understood at
the time of escalation

Customer installs RHEL through proxy; during installation he gets error
#302 from the proxy on package "compat-db".  When he tries to get the
package manually he gets a "too many redirects" error.

   3. State specific action requested of SEG

explain what's causing this and suggest resolution.

   4. State whether or not a defect in the product is suspected

no idea.

Provide supporting info

   1. State other actions already taken in working the problem:

google, fulltext, etc returned nothing relevant        

   2. Attach sosreport

attached (both for proxy and satellite)

   3. Attach other supporting data

satellite debug and proxy debug attached

   4. Provide issue repro information:

unsure

   5. List any known hot-fix packages on the system

none known

   6. List any customer applied changes from the last 30 days 

customer upgraded from 4.2 to 5.0.1 recently.

RHN Specific

   1. State whether Satellite or RHN hosted

satellite/proxy

   2. for Satellite: Provide output of ''satellite-debug'' and
''sysreport'' (for both satellite and proxy).

attached.

   3. If the issue occurred during/after an upgrade:

n/a


Issue escalated to Support Engineering Group by: pvn.
Internal Status set to 'Waiting on SEG'
Status set to: Waiting on Tech

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 8 Issue Tracker 2007-10-17 13:34:05 UTC
Hello Paul,

As per the HTTP status code 302 - The requested resource resides
temporarily under a different URI. Since the redirection might be altered
on occasion, the client SHOULD continue to use the Request-URI for future
requests. This response is only cacheable if indicated by a Cache-Control
or Expires header field.

(http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html)

I've also come across the following bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=175151

Which explains that RHN Proxy should handle redirect request on the
clients behalf, but the issue here is with just one package (compat-db) so
I don't really think that we are affected with the above bugzilla.

As far as the information provided it seems that the 'rhn-proxy-debug -
brspoa24l-rhn-proxy-debug.tar.bz2' is corrupted, I had ask one of the
colleagues to untar it and it failed.

Looking at the access_log from the RHN Proxy sysreport I see the following
entries: 

127.0.0.1 - - [03/Oct/2007:23:02:55 -0300] "GET
/ty-cksm/73e2cee481f60df82339f75b94906c21/RedHat/RPMS/compat-db-4.1.25-9.i386.rpm
HTTP/1.0" 200 890300 "-" "Python-urllib/2.1"
10.109.6.130 - - [03/Oct/2007:23:03:04 -0300] "GET
/ty/94y9C8gN//RedHat/RPMS/compat-db-4.1.25-9.i386.rpm HTTP/1.0" 200
890300 "-" "Python-urllib/2.1"
127.0.0.1 - - [03/Oct/2007:23:03:06 -0300] "HEAD
http://127.0.0.1:80/ty/94y9C8gN//RedHat/RPMS/compat-db-4.1.25-9.x86_64.rpm
HTTP/1.1" 302 0 "-" "Python-urllib/2.1"
10.109.6.130 - - [03/Oct/2007:23:03:06 -0300] "GET
/ty/94y9C8gN//RedHat/RPMS/compat-db-4.1.25-9.x86_64.rpm HTTP/1.0" 302 261
"-" "Python-urllib/2.1"

but I don't see relevant entries for the above in the error_log, the
error_log shows entries till 'Sep 30' and then 'Oct 4', so it seems
that somehow the logs are missing from 1st Oct till 3rd Oct.

I also got to know that the UrlLib module of python doesn't handle 302
response properly (I'll dig more on this one).

Since, RHN Proxy cache requests URI this could be a issue with corrupt
cache headers as well. So I would suggest you to do the following:

1. Clear the RHN Proxy cache and try to reproduce this issue.

A. Stop the following services:
# service rhn-proxy stop

B. Remove the squid cache:
# rm -rf /var/spool/squid/*

C. Create swapping files using this command:
# squid -z

D. Start the services:
# service rhn-proxy start

Now, try and provision a client system and let us know if it fails? While
reproducing this issue please collect the following information.

1. If possible, the anaconda logs from the client system
2. 'rhn-proxy-debug' from the RHN Proxy server (if you edit the file and
add/modify 'debug=5', it will log verbose information)

regards,

Internal Status set to 'Waiting on Support'

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 9 Issue Tracker 2007-10-17 13:34:06 UTC
Paul,

Were you able to find out anything regarding this issue?  We're trying to
get this server provisioned as soon as possible.

Thanks much,

Andy


This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 10 Issue Tracker 2007-10-17 13:34:07 UTC
Hi Andy,

Some updates from engineering that came in this morning:

==========
As per the HTTP status code 302 - The requested resource resides
temporarily under a different URI. Since the redirection might be altered
on occasion, the client SHOULD continue to use the Request-URI for future
requests. This response is only cacheable if indicated by a Cache-Control
or Expires header field.

(http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html)

Looking at the access_log from the RHN Proxy sysreport I see the following
entries:

127.0.0.1 - - [03/Oct/2007:23:02:55 -0300] "GET
/ty-cksm/73e2cee481f60df82339f75b94906c21/RedHat/RPMS/compat-db-4.1.25-9.i386.rpm
HTTP/1.0" 200 890300 "-" "Python-urllib/2.1"
10.109.6.130 - - [03/Oct/2007:23:03:04 -0300] "GET
/ty/94y9C8gN//RedHat/RPMS/compat-db-4.1.25-9.i386.rpm HTTP/1.0" 200
890300 "-" "Python-urllib/2.1"
127.0.0.1 - - [03/Oct/2007:23:03:06 -0300] "HEAD
http://127.0.0.1:80/ty/94y9C8gN//RedHat/RPMS/compat-db-4.1.25-9.x86_64.rpm
HTTP/1.1" 302 0 "-" "Python-urllib/2.1"
10.109.6.130 - - [03/Oct/2007:23:03:06 -0300] "GET
/ty/94y9C8gN//RedHat/RPMS/compat-db-4.1.25-9.x86_64.rpm HTTP/1.0" 302 261
"-" "Python-urllib/2.1"

but I don't see relevant entries for the above in the error_log, the
error_log shows entries till 'Sep 30' and then 'Oct 4', so it seems
that somehow the logs are missing from 1st Oct till 3rd Oct.

Since, RHN Proxy cache requests URI this could be a issue with corrupt
cache headers as well. So I would suggest you to do the following:

1. Clear the RHN Proxy cache and try to reproduce this issue.

A. Stop the following services:
# service rhn-proxy stop

B. Remove the squid cache:
# rm -rf /var/spool/squid/*

C. Create swapping files using this command:
# squid -z

D. Start the services:
# service rhn-proxy start

Now, try and provision a client system and let us know if it fails? While
reproducing this issue please collect the following information.

1. If possible, the anaconda logs from the client system
2. 'rhn-proxy-debug' from the RHN Proxy server (if you edit the file and
add/modify 'debug=5', it will log verbose information)

==========

Internal Status set to 'Waiting on Customer'
Status set to: Waiting on Client
Version changed from '360' to 'UNKNOWN'

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 11 Issue Tracker 2007-10-17 13:34:08 UTC
Paul,

Remove the squid cache didn't help... and I also tried provisioning this
sucker from the RHSS itself... with the same effect.  Ok... see, this
thing is in a **REALLY** remote place... and the network speed is
atrocious... such that... 5 hours into the install.. it's barely 75%
done... 

So, I'm wondering... if the URL that is being used... is it just simply
expiring since the install is taking so long?

This is becoming more critical for me to fix... since my "customer"
needs to get this server into production...

Thanks,

Andy


This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 12 Issue Tracker 2007-10-17 13:34:09 UTC
Hello Paul,

If the issue appears while provisioning from the RHN Satellite directly
then there could be two possible reasons:

1. Either the package is broken on the Satellite server (if the client
failed at the same package 'compat-db' (bypassing RHN Proxy)).
2. There could be a timeout issue.

On the RHN Satellite I see the following:
/etc/rhn/default/rhn_server.conf:

# Session token timeouts
proxy_auth_timeout = 21600.0 # 6 hours

So when the communication begins a session is created which lasts till the
time set in proxy_auth_timeout, so it might happened that it just expired.

So, what I would suggest is:

1. Register a RHEL5 system to the RHN Satellite (through the same RHN
Proxy) and do:
# yum install compat-db

If the above works then the package seems to be fine and its probably a
timeout issue.

2. While provisioning the client, if he skips the compat-db package does
the installation continues?

3. A timeout shouldn't occur if the client is being provisioned from the
RHN Proxy for the second time, cause in the second attempt squid should
cache most of the package headers and the installation will be faster
causing no timeouts. Did he tried provisioning the same client twice from
this RHN Proxy?

4. If the problem still persists, increase the timeout value of
'proxy_auth_timeout' from /etc/rhn/default/rhn_server.conf (RHN
Satellite server)

proxy_auth_timeout = 28800.0 # 8 hours

and restart the 'rhn-satellite' service as:
# service rhn-satellite restart

If there still a issue, please provide us the fresh 'rhn-proxy-debug'
and 'satellite-debug' (I assume all the above tests would be done
keeping 'debug=5' in /etc/rhn/rhn.conf from RHN Proxy and RHN
Satellite).

regards,


Internal Status set to 'Waiting on Support'

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 13 Issue Tracker 2007-10-17 13:34:10 UTC
Hi Andy,

Updated suggestions from engineering.  Let me know if you have any
questions.

==========

If the issue appears while provisioning from the RHN Satellite directly
then there could be two possible reasons:

1. Either the package is broken on the Satellite server (if the client
failed at the same package 'compat-db' (bypassing RHN Proxy)).
2. There could be a timeout issue.

On the RHN Satellite I see the following:
/etc/rhn/default/rhn_server.conf:

# Session token timeouts
proxy_auth_timeout = 21600.0 # 6 hours

So when the communication begins a session is created which lasts till the
time set in proxy_auth_timeout, so it might happened that it just expired.

So, what I would suggest is:

1. Register a RHEL5 system to the RHN Satellite (through the same RHN
Proxy) and do:
# yum install compat-db

If the above works then the package seems to be fine and its probably a
timeout issue.

2. While provisioning the client, if he skips the compat-db package does
the installation continues?

3. A timeout shouldn't occur if the client is being provisioned from the
RHN Proxy for the second time, cause in the second attempt squid should
cache most of the package headers and the installation will be faster
causing no timeouts. Did he tried provisioning the same client twice from
this RHN Proxy?

4. If the problem still persists, increase the timeout value of
'proxy_auth_timeout' from /etc/rhn/default/rhn_server.conf (RHN
Satellite server)

proxy_auth_timeout = 28800.0 # 8 hours

and restart the 'rhn-satellite' service as:
# service rhn-satellite restart

If there still a issue, please provide us the fresh 'rhn-proxy-debug'
and 'satellite-debug' (I assume all the above tests would be done
keeping 'debug=5' in /etc/rhn/rhn.conf from RHN Proxy and RHN
Satellite).

Internal Status set to 'Waiting on Customer'
Status set to: Waiting on Client

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 14 Issue Tracker 2007-10-17 13:34:11 UTC
Paul,

I suspect that the timeout is exactly the issue.  I have increased the
timeout to 12 hours in order to support our REALLY remote servers.  I'll
report back once I've successfully completed the install.

Thanks,

Andy


Internal Status set to 'Waiting on Support'
Status set to: Waiting on Tech

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 15 Issue Tracker 2007-10-17 13:34:12 UTC
Paul,

Ok... so no joy here... I increased the timeout by two-fold... and it
didn't help.  Also, we have attempted the install of this system
multiple times from the same proxy... and it fails on a different
package every time.

Now, I made the change and restarted the RHSS... should I have also
restarted the proxy?

Andy



This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 16 Issue Tracker 2007-10-17 13:34:13 UTC
Paul,

Another thing that doesn't make sense... is that I get the same problem
when trying to provision DIRECTLY from the RHSS... bypassing the proxy
altogether... what would account for that?

Andy



This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 17 Issue Tracker 2007-10-17 13:34:14 UTC
Hi Andy,

Thanks for the updates; I'll pass your info back to engineering.  I
really am stumped here, especially by your news that you're seeing the
same thing when going directly to the satellite.

-- 
Paul Novarese

Internal Status set to 'Waiting on SEG'

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 18 Issue Tracker 2007-10-17 13:34:14 UTC
Hello Paul,

We are debugging this further, meanwhile let us know the following:

Is this the only RHEL5 box that is experiencing this issue? Or there are
other systems too which are behaving in the same manner?
If the problem occurs while the client is connected directly to the
Satellite server then the issue could be on this RHEL5 box itself?

As, the issue persists if the client is connected directly to the
Satellite then we would like to increase the client_auth_timeout as well.
The client_auth_time value plays a role when communicating directly to
satellite.

From the Red Hat Network Satellite server, edit
/etc/rhn/default/rhn_server.conf and set client_auth_timeout to 12 hrs as
well.

Restart the 'rhn-satellite' service. Let us know if the above helps.

Anyways, did the customer grabbed the information asked in my previous
reply, while reproducing?

regards,

Internal Status set to 'Waiting on Support'

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 19 Issue Tracker 2007-10-17 13:34:15 UTC
File uploaded: satellite-debug.tar.bz2

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237
it_file 105461

Comment 20 Issue Tracker 2007-10-17 13:34:16 UTC
Paul,

I tried provisioning DIRECTLY from the RHSS... to no avail.  It seems like
the URL being used by the installer is expiring... I've attached another
debug.

Thanks,

Andy


This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 21 Issue Tracker 2007-10-17 13:34:17 UTC
Andy,

Here's the update from engineering:

As the issue persists if the client is connected directly to the Satellite
then we would like to increase the client_auth_timeout as well. The
client_auth_time value plays a role when communicating directly to
satellite.

From the Red Hat Network Satellite server, edit
/etc/rhn/default/rhn_server.conf and set client_auth_timeout to 12 hrs as
well.

Restart the 'rhn-satellite' service. Let us know if the above helps.

--
Paul Novarese


This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 22 Issue Tracker 2007-10-17 13:34:18 UTC
Paul,

Ok... I can do that... however the current default timeout is one hour
(3600.0).... our install sessions are typically dying at the 4-6 hour
timeframe... so I'm not so certain that this is the issue.  

Andy



This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 23 Issue Tracker 2007-10-17 13:34:19 UTC
Hi Vishal,

I passed the latest update to the customer; however, I'm not sure
increasing client_auth_timeout is going to do much since it's currently
set to 1 hour, but he usually doesn't have problems until 5 or 6 hours
into the install process.

Also, the system having trouble here is installing RHEL4.

Two things the customer really wanted me to stress to you:

1) the system they are currently trying to provision is a very remote
server, with a very slow network connection.  He basically gets one chance
a day to try this, and it's starting to get somewhat critical - if this
goes on too much longer, they will probably have to box the machine up and
ship it to a data center, then ship it back to the remote location.

2) he seems to think that the URL that the system uses to download
packages is expiring.  I think what he's getting at is that each time he
tries to install, the system generates a unique identifier that is
embedded in the URL, and he's wondering if that unique identifier is only
good for a certain period of time.

The customer will go ahead and try your latest suggestion, but I wanted to
flip this back to you to see if any of this information helps.

Thanks

-- 
Paul Novarese

Internal Status set to 'Waiting on SEG'

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 24 Issue Tracker 2007-10-17 13:34:20 UTC
Hello Paul,

Had a discussion over this on IRC with the engineering and I guess the
problem seems to be with the *Slow Network* connection. The "unique
identifier that is embedded in the URL" has a expiry time which is like 4
hrs. But we need to tweak the value from the code.

So, I would suggest you to do the following:

* Login to RHN Satellite server, and backup the file
/usr/lib/perl5/site_perl/5.8.5/RHN/DB/TinyURL.pm.
* Now, edit the above file and on line number 73 make the following
change:

(:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') + 1/6)

to look similar as:

(:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') + 1)

Save the file and restart the RHN Satellite service as:

# service rhn-satellite restart

Let us know if that helps.

regards,

P.S
--- /usr/lib/perl5/site_perl/5.8.5/RHN/DB/TinyURL.pm.patch      2007-10-16
19:22:03.000000000 +0530
+++ /usr/lib/perl5/site_perl/5.8.5/RHN/DB/TinyURL.pm    2007-10-16
19:23:46.000000000 +0530
@@ -70,7 +70,7 @@
 INSERT INTO rhnTinyURL
   (token, url, enabled, expires)
 VALUES
-  (:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') +
1/6)
+  (:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') +
1)
 EOS
   $sth->execute_h(token => $token, url => $url, expires => $expires);
   $dbh->commit;


Internal Status set to 'Waiting on Support'

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 25 Issue Tracker 2007-10-17 13:34:21 UTC
Hi Andy,

After discussion with engineering, they have suggested the following:

==========

The "unique identifier that is embedded in the URL" has a expiry time
which is like 4 hrs. But we need to tweak the value from the code.

So, I would suggest you to do the following:

* Login to RHN Satellite server, and backup the file
/usr/lib/perl5/site_perl/5.8.5/RHN/DB/TinyURL.pm.
* Now, edit the above file and on line number 73 make the following
change:

(:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') + 1/6)

to look similar as:

(:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') + 1)

Save the file and restart the RHN Satellite service as:

# service rhn-satellite restart

Let us know if that helps.

Internal Status set to 'Waiting on Customer'
Status set to: Waiting on Client

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 26 Issue Tracker 2007-10-17 13:34:22 UTC
Paul,

I'm completely at a loss here... this also didn't help... I made this
change and restarted the satellite service... and tried provisioning
from both the proxy and the satellite... and both errored out between 4
and 6 hours into the process....

I don't know what else to suggest at this point.... I'm desperate to
make this work.

Thanks,

Andy 

-----Original Message-----
From: Red Hat Issue Tracker [tao] 
Sent: Tuesday, October 16, 2007 10:08 AM
To: Speagle, Andy - Andy_Speagle
Subject: (UPDATED) Issue #134237 (RHN Proxy Problems)[Cargill]


Update to issue 134237 by pvn
Action: Hi Andy,

After discussion with engineering, they have suggested the following:

==========

The "unique identifier that is embedded in the URL" has a expiry time
which is like 4 hrs. But we need to tweak the value from the code.

So, I would suggest you to do the following:

* Login to RHN Satellite server, and backup the file
/usr/lib/perl5/site_perl/5.8.5/RHN/DB/TinyURL.pm.
* Now, edit the above file and on line number 73 make the following
change:

(:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') + 1/6)

to look similar as:

(:token, :url, 'Y', TO_DATE(:expires, 'YYYY-MM-DD HH24:MI:SS') + 1)

Save the file and restart the RHN Satellite service as:

# service rhn-satellite restart

Let us know if that helps.


Status set to: Waiting on Client


https://enterprise.redhat.com/issue-tracker/134237

Previous Events:
----------------
Posted 10-16-2007 09:33am by Andy_Speagle Paul,

Ok... I can do that... however the current default timeout is one hour
(3600.0).... our install sessions are typically dying at the 4-6 hour
timeframe... so I'm not so certain that this is the issue.  

Andy




Posted 10-16-2007 09:27am by pvn
Andy,

Here's the update from engineering:

As the issue persists if the client is connected directly to the
Satellite then we would like to increase the client_auth_timeout as
well. The client_auth_time value plays a role when communicating
directly to satellite.

From the Red Hat Network Satellite server, edit
/etc/rhn/default/rhn_server.conf and set client_auth_timeout to 12 hrs
as well.

Restart the 'rhn-satellite' service. Let us know if the above helps.

--
Paul Novarese






Internal Status set to 'Waiting on Support'
Status set to: Waiting on Tech

This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 27 Issue Tracker 2007-10-17 13:34:23 UTC
--- Engineering ---

Following are the relevant logs from his satellite-debug:

10.109.6.130 - RHEL Client (which is on a remote *slow* network)
10.117.3.29 - RHN Proxy

access_log:

10.109.6.130 - - [16/Oct/2007:01:56:43 -0500] "GET
/ty/chIuZ6EB//RedHat/RPMS/gimp-print-utils-4.2.7-2.x86_64.rpm HTTP/1.0"
200 21310 "-" "Python-urllib/2.1"
10.109.6.130 - - [16/Oct/2007:01:58:41 -0500] "GET
/ty/chIuZ6EB//RedHat/RPMS/xscreensaver-4.18-5.rhel4.13.x86_64.rpm
HTTP/1.0" 200 6411345 "-" "Python-urllib/2.1"
10.109.6.130 - - [16/Oct/2007:01:58:43 -0500] "GET
/ty/chIuZ6EB//RedHat/RPMS/file-roller-2.8.1-1.x86_64.rpm HTTP/1.0" 302
408 "-" "Python-urllib/2.1"
10.109.6.130 - - [16/Oct/2007:01:58:44 -0500] "GET /errors/404.pxt
HTTP/1.0" 200 3161
10.109.6.130 - - [16/Oct/2007:05:46:56 -0500] "GET
/ty/chIuZ6EB//RedHat/RPMS/file-roller-2.8.1-1.x86_64.rpm HTTP/1.0" 302
408 "-" "Python-urllib/2.1"
10.109.6.130 - - [16/Oct/2007:05:47:02 -0500] "GET /errors/404.pxt
HTTP/1.0" 200 3161


error_log:

[Tue Oct 16 01:58:44 2007] [error] PXT::Request::cookie_jar
(/usr/lib/perl5/site_perl/5.8.5/PXT/Request.pm:410): Generating session
cookie for user: 'none', session name: '
pxt-session-cookie', value: '2345357xbafe7a8c96170e04b3b8a804944ff2bb',
expire: 'never'.
[Tue Oct 16 03:15:10 2007] [error] [client 10.117.3.29] Invalid URI in
request <?xml version='1.0'?>
[Tue Oct 16 05:46:56 2007] [error] PXT::Request::cookie_jar
(/usr/lib/perl5/site_perl/5.8.5/PXT/Request.pm:410): Generating session
cookie for user: 'none', session name: '
pxt-session-cookie', value: '2345358xba6802e01de8a0e0c81d07fae4ad2aae',
expire: 'never'.
[Tue Oct 16 05:47:02 2007] [error] PXT::Request::cookie_jar
(/usr/lib/perl5/site_perl/5.8.5/PXT/Request.pm:410): Generating session
cookie for user: 'none', session name: '
pxt-session-cookie', value: '2345359xa036e1ff48f4fe9303d50e9746494710',
expire: 'never'.


10.117.3.29 - RHN Proxy in question

Could you please suggest as to what could the reason for this timeout and
if a workaround is possible ?

regards,


This event sent from IssueTracker by vgaikwad  [SEG - RHN]
 issue 134237

Comment 28 Miroslav Suchý 2007-10-17 14:28:15 UTC
Changing component to Satellite, as this error happen also when connecting
directly with satellite without proxy, therefore it is not proxy bug.

This RFE https://bugzilla.redhat.com/show_bug.cgi?id=335891 is relevant to this bug.

Comment 29 Issue Tracker 2007-10-17 21:15:51 UTC
Hi Vishal, Miloslav,

I Just spoke with the customer on the phone.  The machine that he's
trying to provision is at a *very* remote site in a South American
country.    If we can't get the install to work by this weekend, he's
going to have to have the computer packed up and shipped to one of his
company's datacenters, have the OS loaded from there, then have the box
packed up again and shipped back to the remote site in order to meet his
deadlines (so, added time and expense, and he's already behind
schedule).

This presents two problems:

1) Can we get a workaround that will allow this to go forward by Friday? 
I know that's short notice, but that's the situation we're faced with
right now.

2) Is there any additional data we want to collect?  Once the box is
packed up we're not going to be able to collect any more, so if there's
anything that we might even *possibly* want, we need to get it now.

I see the two cases here are currently flagged for SS 5.2.0 - I don't
think this is going to be an acceptable timeline for the customer, since
he will be doing installations at these types of extremely remote
locations more and more frequently.  Is this something that will be
addressable in a hotfix?

Thanks,

-- 
Paul Novarese



This event sent from IssueTracker by pvn 
 issue 134237

Comment 30 Miroslav Suchý 2007-10-18 07:33:21 UTC
1) Is the kickstart somehowe special (customer modified)? If not, he can download 
RHEL iso and install it from CD. If he have special KS, shiping computer (well
maybe HDD) seems to me as fastest way.
Oh maybe he cat install it on some computer in his datacenter and then rsync it
to  SouthAmerica and dd copy it to HDD. But this is quite advanced hacking.

2) Can you attach output of satellite-debug to this BZ? This should be enough
for us.

3) Hotfix is not possible. Hotfix can be done only for bug for which we already
know solution. And we do not know sollution for this yet. But you can urge PM to
push it into 5.1 as blocker or give it higher priority.

Comment 31 Issue Tracker 2007-10-18 09:22:28 UTC
Hello Miroslav,

satellite-debug is attached please check "Comment #6 From Issue Tracker"
on the BZ.




This event sent from IssueTracker by vgaikwad 
 issue 134237

Comment 35 Miroslav Suchý 2007-10-31 13:28:37 UTC
Notes for myself:
from #8:
127.0.0.1 - - [03/Oct/2007:23:03:06 -0300] "HEAD
http://127.0.0.1:80/ty/94y9C8gN//RedHat/RPMS/compat-db-4.1.25-9.x86_64.rpm
There is missing session id    ^^ here.

Vishal or Paul,
Does this happend only on RHEL4 (under up2date) or client tried RHEL5 too (with yum)
I know you asked this question, but I can not find answer in comments. 

I'll try to criplle my network and hopefully got reproducer.

Comment 36 Issue Tracker 2007-11-01 21:56:24 UTC
Hi Miroslav,

The customer has not done any RHEL5 deployments, only RHEL4.

Thanks.

-- 
Paul Novarese


This event sent from IssueTracker by pvn 
 issue 134237

Comment 37 Miroslav Suchý 2007-11-02 15:28:57 UTC
OK, I was able to reproduce it. The problem is indeed in tiny url code. But It
is no longer generated by perl code, but java code.
Problem is in file
./code/src/com/redhat/rhn/domain/common/CommonFactory.java on line 121, where it
is set to 4 hours from now. I'll change it, build packages and test if it helped.

Comment 38 Miroslav Suchý 2007-11-02 16:46:17 UTC
Commited into trunk (rev. 133466).
This commit add new option into rhn.conf
server.satellite.tiny_url_timeout
which say how long (in hour) should by tinyurl (/ty/token) valid.

I now run new kickstart and I will see on monday if it finished.

Comment 39 Miroslav Suchý 2007-11-02 16:48:09 UTC
*** Bug 335891 has been marked as a duplicate of this bug. ***

Comment 40 Miroslav Suchý 2007-11-05 09:22:14 UTC
Well, the KS finished OK through weekend. Thats good.

Comment 41 Miroslav Suchý 2007-11-12 09:30:13 UTC
Moving ON_QA:

Satellite 5.1.0-24 and Proxy 5.1.0-15 Content now available on webqa Channels.

Satellite 5.1.0-24 ISOs are now available as
rhn-satellite-5.1.0-24-redhat-linux-as-* @:

http://barn.rhndev.redhat.com/satellite-isos/devel/satellite-5.1/ 

Comment 44 Miroslav Suchý 2007-11-13 10:20:09 UTC
John, we need release doc for this. Something like:

New option "server.satellite.tiny_url_timeout" in rhn.conf is available. This
option say how long is generated kickstart file valid in hours. Default is 4.

Comment 50 Miroslav Suchý 2007-11-14 09:22:39 UTC
I add the new option to default/rhn_web.conf (rev. 133865).

Comment 51 Miroslav Suchý 2007-11-26 10:41:02 UTC
Wrong directory. Commited again (rev. 134197).

Comment 53 John Ha 2007-11-27 23:26:54 UTC
Thank you, Miroslav. Let me know if my edited version of your release note entry
is satisfactory. Thanks in advance.

 server.satellite.tiny_url_timeout is a new option in the /etc/rhn/rhn.conf file
 that allows a user to configure the amount of time (in hours) a kickstart file is
 valid before it times out. The default value is "4".

Comment 54 Miroslav Suchý 2007-11-28 10:52:37 UTC
John, the text sound good for me.

Comment 55 Miroslav Suchý 2007-11-28 10:59:50 UTC
Paul, I think that 4 hours has been chosen, becouse somebody decided "4 hours
should be enough for everybody".
I can not think about any negative effect if you increase the value.
There can be some issues if you rise it up to weeks or months, but anything in
range from hour to 1-2 days is without problems.

Comment 56 Preethi Thomas 2007-12-04 15:37:11 UTC
Miroslav,

Could you please put in a test plan for this one.

thanks

Comment 57 Miroslav Suchý 2007-12-04 16:25:03 UTC
you can follow #46.
set server.satellite.tiny_url_timeout in /etc/rhn/rhn.conf to different values
(in hours) and request KS file, extract tiny url from there (something like
/ty/2TLqqumM) and run:

while true; do rm -f README; date; wget
http://rlx-1-18.rhndev.redhat.com/ty/2TLqqumM/images/README; sleep 180; done

after some time (set by tiny_url_timeout) you start to get http status code 302
instead 200.

Comment 61 Preethi Thomas 2007-12-12 16:42:00 UTC
verified.

Comment 62 Jeff Weiss 2008-03-27 13:31:43 UTC
Verified on 179

Comment 63 Brandon Perkins 2008-04-02 20:31:25 UTC
Proxy 5.1.0 GA so Closed for Current Release.

Comment 64 Brandon Perkins 2008-06-11 02:03:51 UTC
Removing Bug 336151 blocks bug 446208.