Bug 1124570

Summary:	Tasks are waiting for a free worker while other workers are free
Product:	Red Hat Satellite	Reporter:	Ivan Necas <inecas>
Component:	Pulp	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED NOTABUG	QA Contact:	Katello QA List <katello-qa-list>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	Unspecified	CC:	bbuckingham, bkearney, bmbouter, daviddavis, dkliban, ggainey, igulina, ipanova, mhrivnak, pcreech, rchan, skarmark, ttereshc
Target Milestone:	Unspecified	Keywords:	FutureFeature, Reopened
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:
Clones:	1125323 (view as bug list)		Environment:
Last Closed:	2015-03-06 16:55:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	950746, 1124537, 1125323

Description Ivan Necas 2014-07-29 20:21:54 UTC

Description of problem:
With a lot of concurrent async tasks, it's not that hard to get
to a state, where a short running task on one repo (such as repo publish)
is waiting for a long-running task on a different repo (such as repo sync).

This causes troubles when running repo sync on more repositories while doing some content view work in katello (that consists from many short-running tasks).

How reproducible:
When triggering many async tasks in short period

Steps to Reproduce:
1. 4 workers running
2. trigger 2 synces on large repos (ideally several hours to finish)
3. start triggering many short-tasks (could be syncing on small repo, repo publish etc.), all on different repositories than the original ones

Actual results:

the short tasks get assigned to the workers that perform the long-running 
tasks, there is a situation where 2 workers are free to do some work while the short-running tasks were assigned to the busy workers.

Expected results:

The short-running tasks are handled by the free workers

Additional info:

Comment 2 Brian Bouterse 2014-09-04 15:42:45 UTC

This is being prioritized higher and will implement the design outlined on the wiki [0].

[0]:  https://fedorahosted.org/pulp/wiki/TwoQueueReservedResourceDesign

Comment 3 Brian Bouterse 2014-09-15 20:52:57 UTC

PR available here: https://github.com/pulp/pulp/pull/1158

Comment 4 Brian Bouterse 2014-09-19 14:53:29 UTC

Merged to 2.5-dev, master

Comment 5 Chris Duryee 2014-12-23 20:52:38 UTC

fixed in pulp 2.6.0-0.2.beta

Comment 6 Irina Gulina 2015-01-05 15:35:36 UTC

>> rpm -qa pulp-server
pulp-server-2.6.0-0.2.beta.fc20.noarch

>> systemctl | grep pulp_workerpulp_worker-0.service                                                                                  loaded active running   Pulp Worker #0
pulp_worker-1.service                                                                                  loaded active running   Pulp Worker #1
pulp_worker-2.service                                                                                  loaded active running   Pulp Worker #2
pulp_worker-3.service                                                                                  loaded active running   Pulp Worker #3
pulp_workers.service                                                                                   loaded active exited    Pulp Celery Workers

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates21 --feed http://mirror.vutbr.cz/fedora/updates/21/x86_64/
Successfully created repository [updates21]

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates20 --feed http://mirror.vutbr.cz/fedora/updates/20/x86_64/
Successfully created repository [updates20]

>> pulp-admin -u admin -p admin rpm repo create --repo-id updates19 --feed http://mirror.vutbr.cz/fedora/updates/19/x86_64/
Successfully created repository [updates19]

>> pulp-admin -u admin -p admin rpm repo create --repo-id zoo --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
Successfully created repository [zoo]

>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates19
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates19]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[-]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates20
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates20]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[\]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id updates21
+----------------------------------------------------------------------+
                  Synchronizing Repository [updates21]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[/]
^C
>> pulp-admin -u admin -p admin rpm repo sync run --repo-id zoo
+----------------------------------------------------------------------+
                     Synchronizing Repository [zoo]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed


Task Succeeded



Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
32 of 32 items
... completed

Publishing Delta RPMs
... skipped

Publishing Errata
[==================================================] 100%
4 of 4 items
... completed

Publishing Comps file
[==================================================] 100%
3 of 3 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Generating sqlite files
... skipped

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed


Task Succeeded

>> pulp-admin repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                  updates21
Display Name:        updates21
Description:         None
Content Unit Counts: 

Id:                  updates20
Display Name:        updates20
Description:         None
Content Unit Counts: 

Id:                  updates19
Display Name:        updates19
Description:         None
Content Unit Counts: 

Id:                  zoo
Display Name:        zoo
Description:         None
Content Unit Counts: 
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32

>> pulp-admin tasks list
+----------------------------------------------------------------------+
                                 Tasks
+----------------------------------------------------------------------+

Operations:  sync
Resources:   updates19 (repository)
State:       Running
Start Time:  2015-01-05T15:23:07Z
Finish Time: Incomplete
Task Id:     bee2285a-3fd0-4eb0-b95f-80d5b30b7191

Operations:  sync
Resources:   updates20 (repository)
State:       Running
Start Time:  2015-01-05T15:23:18Z
Finish Time: Incomplete
Task Id:     b8f025b5-7cba-41c6-9fb4-e008077409e1

Operations:  sync
Resources:   updates21 (repository)
State:       Running
Start Time:  2015-01-05T15:23:34Z
Finish Time: Incomplete
Task Id:     99da0fb8-d444-4f82-b918-5f2e2315c422

Operations:  sync
Resources:   zoo (repository)
State:       Successful
Start Time:  2015-01-05T15:23:56Z
Finish Time: 2015-01-05T15:23:57Z
Task Id:     f5380a51-3292-4257-8f13-63f6bbb30258

Operations:  publish
Resources:   zoo (repository)
State:       Successful
Start Time:  2015-01-05T15:23:57Z
Finish Time: 2015-01-05T15:23:58Z
Task Id:     ca2c8f12-6948-4224-99d3-0262999dbe50

Comment 7 Brian Bouterse 2015-02-19 01:18:43 UTC

Moved to https://pulp.plan.io/issues/243

Comment 8 RHEL Program Management 2015-03-04 11:24:20 UTC

Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 9 Bryan Kearney 2015-03-06 16:55:48 UTC

Closing. Looks like the pulp migration re-open.

Comment 10 pulp-infra@redhat.com 2015-04-23 16:40:46 UTC

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.