Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1124570 - Tasks are waiting for a free worker while other workers are free
Summary: Tasks are waiting for a free worker while other workers are free
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: Unspecified
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: sat6-pulp-future 1124537 1125323
TreeView+ depends on / blocked
 
Reported: 2014-07-29 20:21 UTC by Ivan Necas
Modified: 2021-04-06 18:31 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
: 1125323 (view as bug list)
Environment:
Last Closed: 2015-03-06 16:55:48 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 243 0 High CLOSED - CURRENTRELEASE Tasks are waiting for a free worker while other workers are free Never

Description Ivan Necas 2014-07-29 20:21:54 UTC 
Description of problem:
With a lot of concurrent async tasks, it's not that hard to get
to a state, where a short running task on one repo (such as repo publish)
is waiting for a long-running task on a different repo (such as repo sync).

This causes troubles when running repo sync on more repositories while doing some content view work in katello (that consists from many short-running tasks).

How reproducible:
When triggering many async tasks in short period

Steps to Reproduce:
1. 4 workers running
2. trigger 2 synces on large repos (ideally several hours to finish)
3. start triggering many short-tasks (could be syncing on small repo, repo publish etc.), all on different repositories than the original ones

Actual results:

the short tasks get assigned to the workers that perform the long-running 
tasks, there is a situation where 2 workers are free to do some work while the short-running tasks were assigned to the busy workers.

Expected results:

The short-running tasks are handled by the free workers

Additional info:
Comment 2 Brian Bouterse 2014-09-04 15:42:45 UTC 
This is being prioritized higher and will implement the design outlined on the wiki [0].

[0]:  https://fedorahosted.org/pulp/wiki/TwoQueueReservedResourceDesign
Comment 3 Brian Bouterse 2014-09-15 20:52:57 UTC 
PR available here: https://github.com/pulp/pulp/pull/1158
Comment 4 Brian Bouterse 2014-09-19 14:53:29 UTC 
Merged to 2.5-dev, master
Comment 5 Chris Duryee 2014-12-23 20:52:38 UTC 
fixed in pulp 2.6.0-0.2.beta
Comment 6 Irina Gulina 2015-01-05 15:35:36 UTC 
>> rpm -qa pulp-server
pulp-server-2.6.0-0.2.beta.fc20.noarch

>> systemctl | grep pulp_workerpulp_worker-0.service                                                                                  loaded active running   Pulp Worker #0
pulp_worker-1.service                                                                                  loaded active running   Pulp Worker #1
pulp_worker-2.service                                                                                  loaded active running   Pulp Worker #2
pulp_worker-3.service                                                                                  loaded active running   Pulp Worker #3
pulp_workers.service                                                                                   loaded active exited    Pulp Celery Workers

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates21 --feed http://mirror.vutbr.cz/fedora/updates/21/x86_64/
Successfully created repository [updates21]

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates20 --feed http://mirror.vutbr.cz/fedora/updates/20/x86_64/
Successfully created repository [updates20]

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates19 --feed http://mirror.vutbr.cz/fedora/updates/19/x86_64/
Successfully created repository [updates19]

>> pulp-admin -u admin -p admin rpm repo create --repo-id zoo --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
Successfully created repository [zoo]

>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates19
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates19]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[-]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates20
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates20]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[\]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates21
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates21]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[/]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id zoo
+----------------------------------------------------------------------+
                     Synchronizing Repository [zoo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed


Task Succeeded



Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed


Task Succeeded

>> pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                  updates21
Display Name:        updates21
Description:         None
Content Unit Counts: 

Id:                  updates20
Display Name:        updates20
Description:         None
Content Unit Counts: 

Id:                  updates19
Display Name:        updates19
Description:         None
Content Unit Counts: 

Id:                  zoo
Display Name:        zoo
Description:         None
Content Unit Counts: 
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32

>> pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  sync
Resources:   updates19 (repository)
State:       Running
Start Time:  2015-01-05T15:23:07Z
Finish Time: Incomplete
Task Id:     bee2285a-3fd0-4eb0-b95f-80d5b30b7191

Operations:  sync
Resources:   updates20 (repository)
State:       Running
Start Time:  2015-01-05T15:23:18Z
Finish Time: Incomplete
Task Id:     b8f025b5-7cba-41c6-9fb4-e008077409e1

Operations:  sync
Resources:   updates21 (repository)
State:       Running
Start Time:  2015-01-05T15:23:34Z
Finish Time: Incomplete
Task Id:     99da0fb8-d444-4f82-b918-5f2e2315c422

Operations:  sync
Resources:   zoo (repository)
State:       Successful
Start Time:  2015-01-05T15:23:56Z
Finish Time: 2015-01-05T15:23:57Z
Task Id:     f5380a51-3292-4257-8f13-63f6bbb30258

Operations:  publish
Resources:   zoo (repository)
State:       Successful
Start Time:  2015-01-05T15:23:57Z
Finish Time: 2015-01-05T15:23:58Z
Task Id:     ca2c8f12-6948-4224-99d3-0262999dbe50
Comment 7 Brian Bouterse 2015-02-19 01:18:43 UTC 
Moved to https://pulp.plan.io/issues/243
Comment 8 RHEL Program Management 2015-03-04 11:24:20 UTC 
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.
Comment 9 Bryan Kearney 2015-03-06 16:55:48 UTC 
Closing. Looks like the pulp migration re-open.
Comment 10 pulp-infra@redhat.com 2015-04-23 16:40:46 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Note You need to log in before you can comment on or make changes to this bug.