Bug 695500 - CDS sync may fail with timeout when syncing takes longer than 90 seconds
Summary: CDS sync may fail with timeout when syncing takes longer than 90 seconds
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: nodes
Version: unspecified
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: Sprint 22
Assignee: Jeff Ortel
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks: verified-to-close
TreeView+ depends on / blocked
 
Reported: 2011-04-11 21:27 UTC by Jeff Ortel
Modified: 2011-08-16 12:22 UTC (History)
1 user (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2011-08-16 12:10:07 UTC
Embargoed:


Attachments (Terms of Use)

Description Jeff Ortel 2011-04-11 21:27:53 UTC
Description of problem:

The CDS dispatcher uses gofer default timeout of 90 seconds for synchronous RMI.  This means that any RMI (like CDS sync()) will return/raise RequestTimout exceptions when it takes longer than 90 seconds to complete.

The code in cds/dispatcher.py should be updated to set timeout as:

(10, 90) initialize()
(10, 90) release()
(10, 7200) sync()
(10, 90) set_global_repo_auth()
(10, 90) set_repo_auth()

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.  register a CDS
2.  Associate 1+ very large repo
3.  sync the CDS
  
Actual results:

Should raise RequestTimeout

Expected results:


Additional info:

Comment 1 Jeff Ortel 2011-04-12 15:55:55 UTC
I think this needs to be asynchronous RMI using an AgentTask.  By doing this synchronously, we tie up the task queue thread for the duration of the sync() on the CDS.  This can lead to the pulp server not servicing other requests properly.  We'd need to do this while preserving the limitation of invoking one sync() at a time on a CDS.

The AgentTask is used to install packages on the consumers and works like this:

AgentTask.execute() --> RMI --> CDS.sync()
AgentTask.succeeded() <-- (ReplyHandler) <-- CDS.sync()

The ReplyHandler does a task lookup using information round-tripped in RMI and invokes either succeeded() of failed() on the task.

Comment 2 Jeff Ortel 2011-04-19 18:49:18 UTC
build: 0.168

Comment 3 Preethi Thomas 2011-05-11 18:16:17 UTC
verified
[root@preethi unit]# rpm -q pulp
pulp-0.0.174-1.fc14.noarch

Ran a very long sync and it completed successfully.

[root@preethi unit]# pulp-admin cds sync --hostname=pulp-cds.usersys.redhat.com
Sync for CDS [pulp-cds.usersys.redhat.com] started
Use "cds status" to check on the progress

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 15:40:23      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 17:58:18      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	running                  
Start Time          	2011-05-11 17:47:53      
Finish Time         	In Progress              

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 15:40:23      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 18:07:07      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	running                  
Start Time          	2011-05-11 17:47:53      
Finish Time         	In Progress              

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 15:40:23      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 18:13:25      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	running                  
Start Time          	2011-05-11 17:47:53      
Finish Time         	In Progress              

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 18:18:30      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 18:21:39      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	finished                 
Start Time          	2011-05-11 17:47:53      
Finish Time         	2011-05-11 18:18:30

Comment 4 Preethi Thomas 2011-08-16 12:10:07 UTC
Closing with Community Release 15

pulp-0.0.223-4.

Comment 5 Preethi Thomas 2011-08-16 12:22:15 UTC
Closing with Community Release 15

pulp-0.0.223-4.


Note You need to log in before you can comment on or make changes to this bug.