Bug 715091

Also, the task list output needs to show arguments to the calls. Otherwise, it's impossible to differentiate between sync calls:

Task: 366c15b3-9c44-11e0-bccb-00508d977dff
    Scheduler: immediate
    Call: CdsApi.cds_sync
    State: running
    Start time: 2011-06-21T16:22:40-04:00
    Finish time: None
    Scheduled time: 2011-06-21T20:22:40Z
    Result: None
    Exception: None
    Traceback: None


The argument to cds_sync identifies the CDS being syncced.

One more note: I was unable to use the task CLI commands to kill the task. The "remove" command had no effect. I had to shut down apache, delete the contents of task_snapshot, and restart it.

Since I had the task ID from the task list command, I probably could have just deleted that row, but with the mongo shell issues we've been seeing it was simpler to just kill the whole collection.

added a task cancel rest api and cli command to cancel any task in a generic manner
dangerous, but can possibly unstick a bad situation
pushed in 1be66992a2e337c6cb3399ef3324ac7eac5e2939

build: 0.198

In order to truly unstick a task, the following steps must be tasken:

1) delete the task snapshot (if one exists)
2) remove the task
3) cancel the task

This will cause the task to forcefully go away without the tasking system trying to restart it.

fails_qa
[root@preethi ~]# rpm -q pulp
pulp-0.0.212-1.fc14.noarch

here is the use case I followed

1. Ran CDS sync.
2. While cds sync was running stopped goferd on the cds
3. Restarted pulp-cds
4. Ran cds sync 
     [root@preethi ~]# pulp-admin cds sync --hostname=pulp-cds.usersys.redhat.com
error: operation failed: Sync already in process for CDS [pulp-cds.usersys.redhat.com]

4. Ran task list
[root@preethi ~]# pulp-admin task list
Task: 73d0dc59-b3aa-11e0-9f87-002564a85a58
    Scheduler: interval
    Call: cull_history
    Arguments: 
    State: waiting
    Start time: None
    Finish time: None
    Scheduled time: 2011-07-22T05:00:00Z
    Result: None
    Exception: None
    Traceback: None

Task: 73d0c466-b3aa-11e0-9f86-002564a85a58
    Scheduler: interval
    Call: cull_audited_events
    Arguments: 
    State: waiting
    Start time: None
    Finish time: None
    Scheduled time: 2011-07-22T01:00:00Z
    Result: None
    Exception: None
    Traceback: None

Task: 80558a63-b3aa-11e0-b191-002564a85a58
    Scheduler: immediate
    Call: CdsApi.cds_sync
    Arguments: pulp-cds.usersys.redhat.com
    State: running
    Start time: 2011-07-21T11:02:50-04:00
    Finish time: None
    Scheduled time: 2011-07-21T15:02:50Z
    Result: None
    Exception: None
    Traceback: None



 
5. Delete task snapshot

[root@preethi ~]# pulp-admin task delete_snapshot --id=80558a63-b3aa-11e0-b191-002564a85a58
Snapshot for task [80558a63-b3aa-11e0-b191-002564a85a58] deleted

6. Remove task

[root@preethi ~]# pulp-admin task remove --id=80558a63-b3aa-11e0-b191-002564a85a58
Task [80558a63-b3aa-11e0-b191-002564a85a58] set for removal


7. Task cancel

[root@preethi ~]# pulp-admin task cancel --id=80558a63-b3aa-11e0-b191-002564a85a58
Task [80558a63-b3aa-11e0-b191-002564a85a58] canceled

8. Run task list and see it hanging

From pulp.log


2011-07-21 11:04:55,033 28824:140175678088960: pulp.server.tasking.task:WARNING: task:404 Deprecated base class Task.cancel() called for [Task 80558a63-b3aa-11e0-b191-002564a85a58: CdsApi.cds_sync(pulp-cds.usersys.redhat.com, )]
2011-07-21 11:04:57,739 28824:140175583282944: gofer.messaging.consumer:ERROR: consumer:387 aa8b56b8-9c33-4dde-aa08-3fb997f6b3e6
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 382, in __fetch
    return self.__receiver.fetch(timeout=timeout)
  File "<string>", line 8, in fetch
  File "/usr/lib64/python2.7/threading.py", line 137, in release
    raise RuntimeError("cannot release un-acquired lock")
RuntimeError: cannot release un-acquired lock
2011-07-21 11:04:57,783 28824:140175583282944: pulp.server.api.cds:ERROR: cds:585 CDS threw an error during sync to CDS [pulp-cds.usersys.redhat.com]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/api/cds.py", line 568, in cds_sync
    self.dispatcher.sync(cds, payload)
  File "/usr/lib/python2.7/site-packages/pulp/server/cds/dispatcher.py", line 138, in sync
    self._send(stub.sync, data)
  File "/usr/lib/python2.7/site-packages/pulp/server/cds/dispatcher.py", line 170, in _send
    result = func(*args)
  File "/usr/lib/python2.7/site-packages/gofer/messaging/stub.py", line 71, in __call__
    return self.stub._send(request, opts)
  File "/usr/lib/python2.7/site-packages/gofer/messaging/stub.py", line 142, in _send
    any=opts.any)
  File "/usr/lib/python2.7/site-packages/gofer/messaging/policy.py", line 123, in send
    reader.close()
  File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 316, in close
    self.__receiver.close()
  File "<string>", line 6, in close
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1040, in close
    try:
CdsMethodException
2011-07-21 11:04:57,788 28824:140175583282944: pulp.server.tasking.task:ERROR: task:381 Task failed: Task 80558a63-b3aa-11e0-b191-002564a85a58: CdsApi.cds_sync(pulp-cds.usersys.redhat.com, )
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 330, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/cds.py", line 568, in cds_sync
    self.dispatcher.sync(cds, payload)
  File "/usr/lib/python2.7/site-packages/pulp/server/cds/dispatcher.py", line 138, in sync
    self._send(stub.sync, data)
  File "/usr/lib/python2.7/site-packages/pulp/server/cds/dispatcher.py", line 170, in _send
    result = func(*args)
  File "/usr/lib/python2.7/site-packages/gofer/messaging/stub.py", line 71, in __call__
    return self.stub._send(request, opts)
  File "/usr/lib/python2.7/site-packages/gofer/messaging/stub.py", line 142, in _send
    any=opts.any)
  File "/usr/lib/python2.7/site-packages/gofer/messaging/policy.py", line 123, in send
    reader.close()
  File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 316, in close
    self.__receiver.close()
  File "<string>", line 6, in close
  File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1040, in close
    try:
PulpException: 'Error on the CDS during sync; check the server log for more information'

CDS code is being rewritten entirely in v2 and this is no longer applicable.