I left my Summit demo environment running over about 5 days. I'm not 100% sure when it happened. It looks like it was when I created a feedless repo, but I wouldn't think that would touch the tasking subsystem (I'm not 100% sure on that though). 2011-05-05 17:06:23,425 [INFO][Dummy-5] _create_repo() @ util.py:364 - started repo metadata update 2011-05-05 17:06:23,460 [CRITICAL][Thread-3] _dispatch() @ queue.py:121 - Exception in FIFO Queue Dispatch Thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/queue.py", line 115, in _dispatch self.run(task) File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/queue.py", line 253, in run self.__storage.store_running(task) File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/storage.py", line 136, in store_running assert task not in self.__complete_tasks AssertionError 2011-05-05 17:06:
Not sure if this is related. But with the latest build (179) I have a repo with schedule and seeing this 2011-05-23 10:00:05,358 [ERROR][Thread-39] _complete() @ task.py:408 - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 406, in _complete self.complete_callback(self) File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/queue.py", line 277, in complete self.__storage.store_complete(task) File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/storage.py", line 143, in store_complete assert task not in self.__waiting_tasks AssertionError
Looks like an additional traceback here. For some reason we're getting out of sync in the storage with scheduled tasks. 2011-05-23 13:02:22,846 [ERROR][Thread-7] _complete() @ task.py:408 - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 406, in _complete self.complete_callback(self) File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/queue.py", line 269, in complete self.__storage.remove_running(task) File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/storage.py", line 139, in remove_running assert task in self.__running_tasks AssertionError
Python falls back to the custom __cmp__ method for an instance if it is defined in the class for testing equality and membership. This is great if you're trying to sort a list by, say, a Task's scheduled_time. It's not great if you're trying to determine if a Task is in a list that contains a Task with the same scheduled_time. However, Python will use a custom __eq__ method for equality and membership before the custom __cmp__, if it present. Bug fix includes added a custom __eq__ function to Task that only returns True if the ids match (in other words, if it's the same damn Task) Fix pushed up in hash: 7a6f84006a263fd77eb99439e7290b955d9b208d
build: 0.180
I have had multiple scheduled syncs running over nights and have not seen this issue since the fix. moving to verified [root@preethi ~]# rpm -q pulp pulp-0.0.181-1.fc14.noarch [root@preethi ~]#
Closing with Community Release 15 pulp-0.0.223-4.