Bug 1124570 - Tasks are waiting for a free worker while other workers are free
Summary: Tasks are waiting for a free worker while other workers are free
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: Unspecified
Hardware: Unspecified
OS: Unspecified
high
unspecified vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: sat6-pulp-future 1124537 1125323
TreeView+ depends on / blocked
 
Reported: 2014-07-29 20:21 UTC by Ivan Necas
Modified: 2018-09-19 15:23 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
: 1125323 (view as bug list)
Environment:
Last Closed: 2015-03-06 16:55:48 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Pulp Redmine 243 High CLOSED - CURRENTRELEASE Tasks are waiting for a free worker while other workers are free Never

Description Ivan Necas 2014-07-29 20:21:54 UTC
Description of problem:
With a lot of concurrent async tasks, it's not that hard to get
to a state, where a short running task on one repo (such as repo publish)
is waiting for a long-running task on a different repo (such as repo sync).

This causes troubles when running repo sync on more repositories while doing some content view work in katello (that consists from many short-running tasks).

How reproducible:
When triggering many async tasks in short period

Steps to Reproduce:
1. 4 workers running
2. trigger 2 synces on large repos (ideally several hours to finish)
3. start triggering many short-tasks (could be syncing on small repo, repo publish etc.), all on different repositories than the original ones

Actual results:

the short tasks get assigned to the workers that perform the long-running 
tasks, there is a situation where 2 workers are free to do some work while the short-running tasks were assigned to the busy workers.

Expected results:

The short-running tasks are handled by the free workers

Additional info:

Comment 2 Brian Bouterse 2014-09-04 15:42:45 UTC
This is being prioritized higher and will implement the design outlined on the wiki [0].

[0]:  https://fedorahosted.org/pulp/wiki/TwoQueueReservedResourceDesign

Comment 3 Brian Bouterse 2014-09-15 20:52:57 UTC
PR available here: https://github.com/pulp/pulp/pull/1158

Comment 4 Brian Bouterse 2014-09-19 14:53:29 UTC
Merged to 2.5-dev, master

Comment 5 Chris Duryee 2014-12-23 20:52:38 UTC
fixed in pulp 2.6.0-0.2.beta

Comment 6 Irina Gulina 2015-01-05 15:35:36 UTC
>> rpm -qa pulp-server
pulp-server-2.6.0-0.2.beta.fc20.noarch

>> systemctl | grep pulp_workerpulp_worker-0.service                                                                                  loaded active running   Pulp Worker #0
pulp_worker-1.service                                                                                  loaded active running   Pulp Worker #1
pulp_worker-2.service                                                                                  loaded active running   Pulp Worker #2
pulp_worker-3.service                                                                                  loaded active running   Pulp Worker #3
pulp_workers.service                                                                                   loaded active exited    Pulp Celery Workers

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates21 --feed http://mirror.vutbr.cz/fedora/updates/21/x86_64/
Successfully created repository [updates21]

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates20 --feed http://mirror.vutbr.cz/fedora/updates/20/x86_64/
Successfully created repository [updates20]

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates19 --feed http://mirror.vutbr.cz/fedora/updates/19/x86_64/
Successfully created repository [updates19]

>> pulp-admin -u admin -p admin rpm repo create --repo-id zoo --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
Successfully created repository [zoo]

>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates19
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates19]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[-]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates20
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates20]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[\]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates21
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates21]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[/]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id zoo
+----------------------------------------------------------------------+
                     Synchronizing Repository [zoo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed


Task Succeeded



Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed


Task Succeeded

>> pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                  updates21
Display Name:        updates21
Description:         None
Content Unit Counts: 

Id:                  updates20
Display Name:        updates20
Description:         None
Content Unit Counts: 

Id:                  updates19
Display Name:        updates19
Description:         None
Content Unit Counts: 

Id:                  zoo
Display Name:        zoo
Description:         None
Content Unit Counts: 
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32

>> pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  sync
Resources:   updates19 (repository)
State:       Running
Start Time:  2015-01-05T15:23:07Z
Finish Time: Incomplete
Task Id:     bee2285a-3fd0-4eb0-b95f-80d5b30b7191

Operations:  sync
Resources:   updates20 (repository)
State:       Running
Start Time:  2015-01-05T15:23:18Z
Finish Time: Incomplete
Task Id:     b8f025b5-7cba-41c6-9fb4-e008077409e1

Operations:  sync
Resources:   updates21 (repository)
State:       Running
Start Time:  2015-01-05T15:23:34Z
Finish Time: Incomplete
Task Id:     99da0fb8-d444-4f82-b918-5f2e2315c422

Operations:  sync
Resources:   zoo (repository)
State:       Successful
Start Time:  2015-01-05T15:23:56Z
Finish Time: 2015-01-05T15:23:57Z
Task Id:     f5380a51-3292-4257-8f13-63f6bbb30258

Operations:  publish
Resources:   zoo (repository)
State:       Successful
Start Time:  2015-01-05T15:23:57Z
Finish Time: 2015-01-05T15:23:58Z
Task Id:     ca2c8f12-6948-4224-99d3-0262999dbe50

Comment 7 Brian Bouterse 2015-02-19 01:18:43 UTC
Moved to https://pulp.plan.io/issues/243

Comment 8 RHEL Product and Program Management 2015-03-04 11:24:20 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 9 Bryan Kearney 2015-03-06 16:55:48 UTC
Closing. Looks like the pulp migration re-open.

Comment 10 pulp-infra@redhat.com 2015-04-23 16:40:46 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.


Note You need to log in before you can comment on or make changes to this bug.