Bug 695500

Summary:	CDS sync may fail with timeout when syncing takes longer than 90 seconds
Product:	[Retired] Pulp	Reporter:	Jeff Ortel <jortel>
Component:	nodes	Assignee:	Jeff Ortel <jortel>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	unspecified	CC:	jason.dobies
Target Milestone:	---	Keywords:	Triaged
Target Release:	Sprint 22
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-08-16 12:10:07 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	647488

Description Jeff Ortel 2011-04-11 21:27:53 UTC

Description of problem:

The CDS dispatcher uses gofer default timeout of 90 seconds for synchronous RMI.  This means that any RMI (like CDS sync()) will return/raise RequestTimout exceptions when it takes longer than 90 seconds to complete.

The code in cds/dispatcher.py should be updated to set timeout as:

(10, 90) initialize()
(10, 90) release()
(10, 7200) sync()
(10, 90) set_global_repo_auth()
(10, 90) set_repo_auth()

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.  register a CDS
2.  Associate 1+ very large repo
3.  sync the CDS
  
Actual results:

Should raise RequestTimeout

Expected results:


Additional info:

Comment 1 Jeff Ortel 2011-04-12 15:55:55 UTC

I think this needs to be asynchronous RMI using an AgentTask.  By doing this synchronously, we tie up the task queue thread for the duration of the sync() on the CDS.  This can lead to the pulp server not servicing other requests properly.  We'd need to do this while preserving the limitation of invoking one sync() at a time on a CDS.

The AgentTask is used to install packages on the consumers and works like this:

AgentTask.execute() --> RMI --> CDS.sync()
AgentTask.succeeded() <-- (ReplyHandler) <-- CDS.sync()

The ReplyHandler does a task lookup using information round-tripped in RMI and invokes either succeeded() of failed() on the task.

Comment 2 Jeff Ortel 2011-04-19 18:49:18 UTC

build: 0.168

Comment 3 Preethi Thomas 2011-05-11 18:16:17 UTC

verified
[root@preethi unit]# rpm -q pulp
pulp-0.0.174-1.fc14.noarch

Ran a very long sync and it completed successfully.

[root@preethi unit]# pulp-admin cds sync --hostname=pulp-cds.usersys.redhat.com
Sync for CDS [pulp-cds.usersys.redhat.com] started
Use "cds status" to check on the progress

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 15:40:23      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 17:58:18      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	running                  
Start Time          	2011-05-11 17:47:53      
Finish Time         	In Progress              

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 15:40:23      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 18:07:07      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	running                  
Start Time          	2011-05-11 17:47:53      
Finish Time         	In Progress              

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 15:40:23      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 18:13:25      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	running                  
Start Time          	2011-05-11 17:47:53      
Finish Time         	In Progress              

[root@preethi unit]# pulp-admin cds status --hostname=pulp-cds.usersys.redhat.com
+------------------------------------------+
                 CDS Status
+------------------------------------------+

Name                	pulp-cds.usersys.redhat.com
Hostname            	pulp-cds.usersys.redhat.com
Description         	None                     
Repos               	repos-pulp-pulp-fedora-14-x86_64, f14-update
Last Sync           	2011-05-11 18:18:30      
Status:
   Responding       	Yes                      
   Last Heartbeat   	2011-05-11 18:21:39      

+------------------------------------------+
           Most Recent Sync Tasks
+------------------------------------------+

State               	finished                 
Start Time          	2011-05-11 17:47:53      
Finish Time         	2011-05-11 18:18:30

Comment 4 Preethi Thomas 2011-08-16 12:10:07 UTC

Closing with Community Release 15

pulp-0.0.223-4.

Comment 5 Preethi Thomas 2011-08-16 12:22:15 UTC

Closing with Community Release 15

pulp-0.0.223-4.