Description of problem: With a lot of concurrent async tasks, it's not that hard to get to a state, where a short running task on one repo (such as repo publish) is waiting for a long-running task on a different repo (such as repo sync). This causes troubles when running repo sync on more repositories while doing some content view work in katello (that consists from many short-running tasks). How reproducible: When triggering many async tasks in short period Steps to Reproduce: 1. 4 workers running 2. trigger 2 synces on large repos (ideally several hours to finish) 3. start triggering many short-tasks (could be syncing on small repo, repo publish etc.), all on different repositories than the original ones Actual results: the short tasks get assigned to the workers that perform the long-running tasks, there is a situation where 2 workers are free to do some work while the short-running tasks were assigned to the busy workers. Expected results: The short-running tasks are handled by the free workers Additional info:
This is being prioritized higher and will implement the design outlined on the wiki [0]. [0]: https://fedorahosted.org/pulp/wiki/TwoQueueReservedResourceDesign
PR available here: https://github.com/pulp/pulp/pull/1158
Merged to 2.5-dev, master
fixed in pulp 2.6.0-0.2.beta
>> rpm -qa pulp-server pulp-server-2.6.0-0.2.beta.fc20.noarch >> systemctl | grep pulp_workerpulp_worker-0.service loaded active running Pulp Worker #0 pulp_worker-1.service loaded active running Pulp Worker #1 pulp_worker-2.service loaded active running Pulp Worker #2 pulp_worker-3.service loaded active running Pulp Worker #3 pulp_workers.service loaded active exited Pulp Celery Workers >> pulp-admin -u admin -p admin rpm repo create --repo-id updates21 --feed http://mirror.vutbr.cz/fedora/updates/21/x86_64/ Successfully created repository [updates21] >> pulp-admin -u admin -p admin rpm repo create --repo-id updates20 --feed http://mirror.vutbr.cz/fedora/updates/20/x86_64/ Successfully created repository [updates20] >> pulp-admin -u admin -p admin rpm repo create --repo-id updates19 --feed http://mirror.vutbr.cz/fedora/updates/19/x86_64/ Successfully created repository [updates19] >> pulp-admin -u admin -p admin rpm repo create --repo-id zoo --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/ Successfully created repository [zoo] >> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates19 +----------------------------------------------------------------------+ Synchronizing Repository [updates19] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [-] ^C >> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates20 +----------------------------------------------------------------------+ Synchronizing Repository [updates20] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [\] ^C >> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates21 +----------------------------------------------------------------------+ Synchronizing Repository [updates21] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [/] ^C >> pulp-admin -u admin -p admin rpm repo sync run --repo-id zoo +----------------------------------------------------------------------+ Synchronizing Repository [zoo] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [|] ... completed Downloading repository content... [==================================================] 100% RPMs: 0/0 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [-] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 32 of 32 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 4 of 4 items ... completed Publishing Comps file [==================================================] 100% 3 of 3 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [-] ... completed Writing Listings File [-] ... completed Task Succeeded >> pulp-admin repo list +----------------------------------------------------------------------+ Repositories +----------------------------------------------------------------------+ Id: updates21 Display Name: updates21 Description: None Content Unit Counts: Id: updates20 Display Name: updates20 Description: None Content Unit Counts: Id: updates19 Display Name: updates19 Description: None Content Unit Counts: Id: zoo Display Name: zoo Description: None Content Unit Counts: Erratum: 4 Package Category: 1 Package Group: 2 Rpm: 32 >> pulp-admin tasks list +----------------------------------------------------------------------+ Tasks +----------------------------------------------------------------------+ Operations: sync Resources: updates19 (repository) State: Running Start Time: 2015-01-05T15:23:07Z Finish Time: Incomplete Task Id: bee2285a-3fd0-4eb0-b95f-80d5b30b7191 Operations: sync Resources: updates20 (repository) State: Running Start Time: 2015-01-05T15:23:18Z Finish Time: Incomplete Task Id: b8f025b5-7cba-41c6-9fb4-e008077409e1 Operations: sync Resources: updates21 (repository) State: Running Start Time: 2015-01-05T15:23:34Z Finish Time: Incomplete Task Id: 99da0fb8-d444-4f82-b918-5f2e2315c422 Operations: sync Resources: zoo (repository) State: Successful Start Time: 2015-01-05T15:23:56Z Finish Time: 2015-01-05T15:23:57Z Task Id: f5380a51-3292-4257-8f13-63f6bbb30258 Operations: publish Resources: zoo (repository) State: Successful Start Time: 2015-01-05T15:23:57Z Finish Time: 2015-01-05T15:23:58Z Task Id: ca2c8f12-6948-4224-99d3-0262999dbe50
Moved to https://pulp.plan.io/issues/243
Since this issue was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.
Closing. Looks like the pulp migration re-open.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.