1082968 – Reaper is not cleaning old tasks

Bug 1082968 - Reaper is not cleaning old tasks

Summary: Reaper is not cleaning old tasks

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Pulp
Classification:	Retired
Component:	API/integration
Sub Component:
Version:	unspecified
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	unspecified
Target Milestone:	---
Target Release:	2.4.0
Assignee:	Sayli Karmarkar
QA Contact:	Ina Panova
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-04-01 08:21 UTC by Ina Panova
Modified:	2015-03-23 01:12 UTC (History)
CC List:	2 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2014-08-09 06:56:19 UTC
Embargoed:

Attachments	(Terms of Use)

Description Ina Panova 2014-04-01 08:21:54 UTC

Description of problem:
Seems that the reaper does not clean old tasks at all, so *all* the tasks are keeping to accumulate.
As an example 'pulp-admin tasks list' shows tasks one month old:

Operations:  delete
Resources:   test (repository)
State:       Successful
Start Time:  1393593281
Finish Time: 1393593281
Result:      N/A
Task Id:     965d14fa-a513-4956-bf37-ecfb6183b159


Version-Release number of selected component (if applicable):
pulp-server-2.4.0-0.3.alpha.git.13.3215df6.fc20.noarch

How reproducible:
always

Steps to Reproduce:
1. pulp-admin tasks list 
2.
3.

Actual results:
all tasks are stored, even those that are one month old

Expected results:
after period of time old tasks should be removed

Additional info:

Comment 1 Sayli Karmarkar 2014-05-01 08:14:58 UTC

https://github.com/pulp/pulp/pull/929

Comment 2 Randy Barlow 2014-05-15 17:00:13 UTC

The fix for this bug is included in the pulp-2.4.0-0.14.beta builds.

Comment 3 Ina Panova 2014-05-26 09:48:29 UTC

1)Task list before changes in data reaper conf:

# pulp-admin -u admin -p admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-24T22:35:23Z
Finish Time: 2014-05-24T22:35:23Z
Task Id:     cb5e3318-b3d9-4568-812d-f92113dd0857

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-25T16:35:23Z
Finish Time: 2014-05-25T16:35:23Z
Task Id:     d9d6881e-2194-4327-b9a9-4d3c9d8b4736

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-25T04:35:23Z
Finish Time: 2014-05-25T04:35:23Z
Task Id:     7f611c08-1f9d-4633-b9e0-a77c05b36452

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-25T10:35:23Z
Finish Time: 2014-05-25T10:35:23Z
Task Id:     8e64b78c-8784-4e7f-9465-a8dfc7ea0c56

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-25T22:35:23Z
Finish Time: 2014-05-25T22:35:24Z
Task Id:     a4558b39-d412-4583-afb3-78ee1e119956


2) Changed reaper conf in /etc/pulp/server.conf

[data_reaping]
reaper_interval: 0.0001
# archived_calls: 0.5
# consumer_history: 60
# repo_sync_history: 60
# repo_publish_history: 60
# repo_group_publish_history: 60
task_status_history: 1
task_result_history: 1


3) Restarted the services.



4) Task list after data reaper conf changes:

[root@ec2-54-246-60-188 ~]# pulp-admin -u admin -p admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:19:47Z
Finish Time: 2014-05-26T09:19:47Z
Task Id:     b5c1c5c7-744e-4328-a9fc-e55db5500770

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:20:30Z
Finish Time: 2014-05-26T09:20:30Z
Task Id:     881b19c3-2b41-40df-a724-5a3752f13bc2

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:20:39Z
Finish Time: 2014-05-26T09:20:39Z
Task Id:     43478440-2a4a-4de3-bca2-2f46f88a3bad

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:20:48Z
Finish Time: 2014-05-26T09:20:48Z
Task Id:     231cbdb3-0809-4f90-b4e1-05737ba77430

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:20:56Z
Finish Time: 2014-05-26T09:20:56Z
Task Id:     61fd9781-e9fa-40a6-a3de-5a90291e9c7e

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-25T16:35:23Z
Finish Time: 2014-05-25T16:35:23Z
Task Id:     d9d6881e-2194-4327-b9a9-4d3c9d8b4736

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:21:14Z
Finish Time: 2014-05-26T09:21:14Z
Task Id:     cba73f11-8889-459a-85b6-72406cb306e4

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-25T10:35:23Z
Finish Time: 2014-05-25T10:35:23Z
Task Id:     8e64b78c-8784-4e7f-9465-a8dfc7ea0c56

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-25T22:35:23Z
Finish Time: 2014-05-25T22:35:24Z
Task Id:     a4558b39-d412-4583-afb3-78ee1e119956

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:19:23Z
Finish Time: 2014-05-26T09:19:23Z
Task Id:     f3e92252-9900-4423-940a-ecf81bef8704

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:19:30Z
Finish Time: 2014-05-26T09:19:30Z
Task Id:     884c74f8-c1dc-4d53-9c6f-36166f7cd30d

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:19:38Z
Finish Time: 2014-05-26T09:19:38Z
Task Id:     89dc6171-c57a-417b-a20e-8162003826a4

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:19:56Z
Finish Time: 2014-05-26T09:19:56Z
Task Id:     45bed6d8-c618-401b-b20f-4336f0093d9e

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:20:04Z
Finish Time: 2014-05-26T09:20:04Z
Task Id:     7f62bfbf-50ec-4a48-a77d-f9bd242ac608

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:20:13Z
Finish Time: 2014-05-26T09:20:13Z
Task Id:     c8a90020-fb55-4ed1-963b-630c794bc7da

Operations:  
Resources:   
State:       Successful
Start Time:  2014-05-26T09:20:22Z
Finish Time: 2014-05-26T09:20:22Z
Task Id:     30b21037-453b-4eac-abce-358754d66b50

[root@ec2-54-246-60-188 ~]# less /var/log/messages | grep pulp.server.db.reaper
May 25 10:35:23 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 25 10:35:23 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[8e64b78c-8784-4e7f-9465-a8dfc7ea0c56]
May 25 10:35:23 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 25 10:35:23 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 25 10:35:23 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[8e64b78c-8784-4e7f-9465-a8dfc7ea0c56] succeeded in 0.0305428429856s: None
May 25 16:35:23 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 25 16:35:23 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[d9d6881e-2194-4327-b9a9-4d3c9d8b4736]
May 25 16:35:23 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 25 16:35:23 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 25 16:35:23 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[d9d6881e-2194-4327-b9a9-4d3c9d8b4736] succeeded in 0.0319467600202s: None
May 25 22:35:23 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 25 22:35:23 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[a4558b39-d412-4583-afb3-78ee1e119956]
May 25 22:35:23 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 25 22:35:24 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 25 22:35:24 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[a4558b39-d412-4583-afb3-78ee1e119956] succeeded in 0.0664272809518s: None
May 26 09:19:21 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:19:23 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[f3e92252-9900-4423-940a-ecf81bef8704]
May 26 09:19:23 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:19:23 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:19:23 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[f3e92252-9900-4423-940a-ecf81bef8704] succeeded in 0.568147872982s: None
May 26 09:19:30 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:19:30 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[884c74f8-c1dc-4d53-9c6f-36166f7cd30d]
May 26 09:19:30 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:19:30 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:19:30 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[884c74f8-c1dc-4d53-9c6f-36166f7cd30d] succeeded in 0.0518197400379s: None
May 26 09:19:38 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:19:38 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[89dc6171-c57a-417b-a20e-8162003826a4]
May 26 09:19:38 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:19:38 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:19:38 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[89dc6171-c57a-417b-a20e-8162003826a4] succeeded in 0.060504886962s: None

-------skipping-----
May 26 09:20:56 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:20:56 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[61fd9781-e9fa-40a6-a3de-5a90291e9c7e]
May 26 09:20:56 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:20:56 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:20:56 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[61fd9781-e9fa-40a6-a3de-5a90291e9c7e] succeeded in 0.0252788400394s: None
May 26 09:21:11 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:21:14 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[cba73f11-8889-459a-85b6-72406cb306e4]
May 26 09:21:14 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:21:14 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:21:14 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[cba73f11-8889-459a-85b6-72406cb306e4] succeeded in 0.254793822998s: None
May 26 09:21:20 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:21:20 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[7649dd18-70fd-4d1d-af45-d195889a30a5]
May 26 09:21:20 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:21:20 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:21:20 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[7649dd18-70fd-4d1d-af45-d195889a30a5] succeeded in 0.035506582004s: None
May 26 09:21:28 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:21:28 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[3504dd61-b6ec-4e73-a6fe-eaf4d597e7ce]
May 26 09:21:28 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:21:28 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:21:28 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[3504dd61-b6ec-4e73-a6fe-eaf4d597e7ce] succeeded in 0.0544633640093s: None
May 26 09:21:37 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:21:37 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[fba6d475-45b2-4a0d-9095-048fc1fec3ae]
May 26 09:21:37 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:21:37 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:21:37 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[fba6d475-45b2-4a0d-9095-048fc1fec3ae] succeeded in 0.0702857300057s: None
May 26 09:21:46 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:21:46 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[e59a9a91-ce1a-44ba-9305-9ab3ed35d17a]
May 26 09:21:46 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:21:46 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:21:46 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[e59a9a91-ce1a-44ba-9305-9ab3ed35d17a] succeeded in 0.0696093949955s: None
May 26 09:21:54 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:21:54 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[e2c43da0-68e8-4c5d-b840-ee89294e6fdd]
May 26 09:21:54 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:21:54 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.
May 26 09:21:54 ec2-54-246-60-188 pulp: celery.worker.job:INFO: Task pulp.server.db.reaper.reap_expired_documents[e2c43da0-68e8-4c5d-b840-ee89294e6fdd] succeeded in 0.026877338998s: None
May 26 09:22:03 ec2-54-246-60-188 pulp: celery.beat:INFO: Scheduler: Sending due task reap_expired_documents (pulp.server.db.reaper.reap_expired_documents)
May 26 09:22:03 ec2-54-246-60-188 pulp: celery.worker.strategy:INFO: Received task: pulp.server.db.reaper.reap_expired_documents[83c05636-fcb0-419f-a681-8109fdeb0827]
May 26 09:22:03 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
May 26 09:22:03 ec2-54-246-60-188 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.


5) Result: Only for the last day(24h) task history should be present in db ----True
   Tested in 2.4.0-0.17.beta
   Moving to Verified.

Comment 4 Randy Barlow 2014-08-09 06:56:19 UTC

This has been fixed in Pulp 2.4.0-1.

Note You need to log in before you can comment on or make changes to this bug.