Bug 729496 - Adding timeout in sync ends with error in log
Summary: Adding timeout in sync ends with error in log
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: user-experience
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: Sprint 28
Assignee: Jason Connor
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-09 22:36 UTC by Preethi Thomas
Modified: 2014-03-31 01:39 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-02-24 20:16:07 UTC


Attachments (Terms of Use)

Description Preethi Thomas 2011-08-09 22:36:29 UTC
Description of problem:
Adding timeout in sync ends with error in log


from the pulp.log 

2011-08-09 16:16:52,151 8132:140569892341504: pulp.server.webservices.controllers.repositories:INFO: repositories:609 sync timeout passed : P2Y3M4DT5H6M
2011-08-09 16:16:52,540 8132:140568155903744: pulp.server.api.repo_sync:INFO: repo_sync:222 Sync invoked for repo <f14>
2011-08-09 16:16:52,596 8132:140569671800576: pulp:CRITICAL: queue:128 Exception in FIFO Queue Dispatch Thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/queue.py", line 124, in _dispatch
    self._timeout_tasks()
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/queue.py", line 183, in _timeout_tasks
    if now - task.start_time < task.timeout_delta:
TypeError: can't compare datetime.timedelta to Duration


After restarting pulp-server got ISE using pulp cli commands

Also saw this in ssl_log
[Tue Aug 09 18:25:57 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7
/site-packages/isodate/duration.py", line 95, in __getattr__
[Tue Aug 09 18:25:57 2011] [error] [client 127.0.0.1]     return getattr(self.td
elta, name)
[Tue Aug 09 18:25:57 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7
/site-packages/isodate/duration.py", line 95, in __getattr__
[Tue Aug 09 18:25:57 2011] [error] [client 127.0.0.1]     return getattr(self.td
elta, name)
[Tue Aug 09 18:25:57 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7
/site-packages/isodate/duration.py", line 95, in __getattr__
[Tue Aug 09 18:25:57 2011] [error] [client 127.0.0.1]     return getattr(self.td
elta, name)
[Tue Aug 09 18:25:57 2011] [error] [client 127.0.0.1] RuntimeError: maximum recu
rsion depth exceeded


Version-Release number of selected component (if applicable):
[root@preethi ~]# rpm -q pulp
pulp-0.0.223-1.fc14.noarch
[root@preethi ~]#

Comment 1 James Slagle 2011-08-10 17:29:38 UTC
To work around the 'maximumrecursion depth exceeded' error, use the mongo shell to clear out the task_snapshots collection:

$ mongo pulp_database
MongoDB shell version: 1.8.2
connecting to: pulp_database
> db.task_snapshots.drop()
true
>

Comment 2 Chris Evich 2011-09-01 20:38:36 UTC
Stumbled across this bug.  How about something like this as a fix:

--- src/pulp/server/tasking/taskqueue/queue.py.original	2011-09-01 16:20:32.000000000 -0400
+++ src/pulp/server/tasking/taskqueue/queue.py.modified	2011-09-01 16:28:07.000000000 -0400
@@ -180,7 +180,11 @@
             # queue, but the task thread has not had a chance to execute yet
             if None in (task.timeout_delta, task.start_time):
                 continue
-            if now - task.start_time < task.timeout_delta:
+            if isinstance(task.timeout_delta, isodate.Duration):
+                timedelta_tasktimeout = task.timeout_delta.todatetime(start=task.start_time)
+            else:
+                timedelta_tasktimeout = task.timeout_delta
+            if now - task.start_time < timedelta_tasktimeout:
                 continue
             task.timeout()

i.e. make sure everything is in a common timedelta format 

Alternatively, validate and throw an exception when setting task.timeout_delta if it's longer than 365 days - realistically, will a task ever need to timeout longer than that?  i.e. completely avoid the 1y != 365d problem.

Comment 3 Jason Connor 2011-09-09 22:43:47 UTC
Added a timeout parsing wrapper to the webservices that disallows timeout intervals to be specified with months and years.

committed in fd38843e56ecd250b46a1ea5a09f2a9eb022b04a

Comment 4 Jeff Ortel 2011-09-14 20:27:04 UTC
build: 0.231

Comment 5 Preethi Thomas 2011-09-20 18:15:58 UTC
fails_qa
[root@preethi ~]# rpm -q pulp
pulp-0.0.232-1.fc15.noarch

failing since the help needs to updated

[root@preethi ~]# pulp-admin -u admin -p admin repo sync --id=f15 --help
Usage: pulp-admin <options> repo sync <options>

Options:
  -h, --help         show this help message and exit
  --id=ID            repository id (required)
  --timeout=TIMEOUT  repository sync timeout specified in iso8601 duration
                     format (P[n]Y[n]M[n]DT[n]H[n]M[n]S)

Comment 6 Jeff Ortel 2011-10-13 00:49:18 UTC
build: 0.238

Comment 7 Preethi Thomas 2011-10-13 19:50:12 UTC
fails_qa

help needs to updated


[root@preethi ~]# rpm -q pulp
pulp-0.0.238-1.fc15.noarch

failing since the help needs to updated

[root@preethi ~]# pulp-admin repo sync --id=f14  --help
Usage: pulp-admin <options> repo sync <options>

Options:
  -h, --help         show this help message and exit
  --id=ID            repository id (required)
  --timeout=TIMEOUT  repository sync timeout specified in iso8601 duration
                     format (P[n]Y[n]M[n]DT[n]H[n]M[n]S)

Comment 8 Jason Connor 2011-10-28 20:25:32 UTC
this has been changed by the sync scheduling stuff.

Comment 9 Jeff Ortel 2011-10-28 23:12:46 UTC
build: 0.244

Comment 10 Preethi Thomas 2011-11-14 20:54:36 UTC
verified
[root@preethi ~]# rpm -q pulp
pulp-0.0.248-1.fc15.noarch


[root@preethi ~]# pulp-admin repo sync --id=f14  --help
Usage: pulp-admin <options> repo sync <options>


  --timeout=TIMEOUT    repository sync timeout specified in iso8601 duration
                       format (P[n]Y[n]M[n]DT[n]H[n]M[n]S)

Comment 11 Preethi Thomas 2012-02-24 20:16:07 UTC
Pulp v1.0 is released
Closed Current Release.

Comment 12 Preethi Thomas 2012-02-24 20:17:34 UTC
Pulp v1.0 is released.


Note You need to log in before you can comment on or make changes to this bug.