Bug 889893
| Summary: | Getting a lot of strange tasks created by a scheduled sync | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Retired] Pulp | Reporter: | Steven Roberts <strobert> | ||||||||
| Component: | async/tasks | Assignee: | Jason Connor <jconnor> | ||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 2.0.6 | CC: | jason.dobies, jortel, mmccune, skarmark | ||||||||
| Target Milestone: | --- | Keywords: | Triaged | ||||||||
| Target Release: | 2.0.7 | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2013-04-08 16:01:16 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Steven Roberts
2012-12-23 21:17:28 UTC
the command I used to create the scheduled task: pulp-admin rpm repo sync schedules create --repo-id rhn-rhels6 --schedule 2012-12-01T11:00Z/P1D repo details: Id: rhn-rhels6 Display Name: rhn-rhels6 Description: None Content Unit Count: 0 Notes: Importers: Config: Feed URL: https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os Skip: drpm Last Sync: None Repo Id: rhn-rhels6 Scheduled Syncs: 50d6b09c6381d3498e0005ad Created attachment 668168 [details]
mongoexport of the *_calls collections
Created attachment 668169 [details]
pulp.log
Created attachment 668170 [details]
output of 'pulp-admin tasks list'
and output of pulp-admin schedule list:
+------------------------------------------------------------------------------------------------+
Schedules
+------------------------------------------------------------------------------------------------+
Schedule: 2012-12-01T11:00Z/P1D
Id: 50d6b09c6381d3498e0005ad
Enabled: True
Remaining Runs: N/A
Consecutive Failures: 0
Failure Threshold: None
First Run: 2012-12-23T11:00:00Z
Last Run: None
Next Run: 2012-12-23T11:00:00Z
Aligning to 2.0.7 so jconnor can take a look this week. fix pushed in branch jconnor-889893-scheduled-sync-collisions I in thoery: - applied the code patch to my local pulp server - cleared existing calls - db.archived_calls.remove() - db.queued_calls.remove() - db.scheduled_calls.remove() - put in a fresh schedule entry that kicked off 15 minutes ago I am at 60 queue_calls and counting. If I manually run db.scheduled_calls.remove() it looks to have stopped the growth. I'm going to run a few more tests this afternoon. see if it is something specific to the RH CDN pulls I have done a few scheduled ones against the pulp repo and it is working okay (I am seeing some non expected behavior as to when it is scheduling the next run when I update the schedule, but that I'll do as a separee bug if it ends up being one). so not sure if it is the client https cert, that it is https or the size of the RH one. I'll see if I can isolate a bit more. build: 2.0.7-0.1.beta installed the beta RPM. looking good. and looks like I didn't grab enough of the patch before (comparing what I applied vs the new scheduler.py). my sync that was causing db.queued_calls.count() to growth large before is now behaving. I am seeing a LOT of the 'Schedule %(s)s skipped: last scheduled call still running' messages. like every 30s. so sounds like the pulp scheduler was trying to fire the task because it wasn't done yet even though it was only schedule to run say once per day. I will let you know if it completes properly -- probably a few hours to finish. Flipping back to ON_QA, looks like bugzilla didn't resolve its "mid-air collision" correctly. Moving to verified
[root@pulp-v2-testing ~]# rpm -q pulp-server
pulp-server-2.0.7-0.1.beta.noarch
[root@pulp-v2-testing ~]#
[root@pulp-v2-testing ~]# pulp-admin rpm repo sync schedules list --repo-id rhn-rhels6
+----------------------------------------------------------------------+
Schedules
+----------------------------------------------------------------------+
Schedule: 2013-01-15T09:00Z/P1D
Id: 50f575804a4d38671b0000dd
Enabled: True
Last Run: 2013-01-15T15:30:00Z
Next Run: 2013-01-16T15:30:00Z
[root@pulp-v2-testing ~]# pulp-admin tasks list
+----------------------------------------------------------------------+
Tasks
+----------------------------------------------------------------------+
Operations: profile_create
Resources: f17-client (consumer), rpm (content_unit)
State: Successful
Start Time: 2013-01-16T08:50:30Z
Finish Time: 2013-01-16T08:50:30Z
Result: ****
Task Id: 02fee600-38c4-4920-87bc-89299cd68e2c
[root@pulp-v2-testing ~]#
Agree on the verified... fyi though the key is that when the scheduled task runs it doesn't end up making thousands of tasks not just when the schedule gets created. Pulp 2.1 released http://www.pulpproject.org/2013/04/05/pulp-2-1-0-released/ |