1986356 – pulp3: independent CV publishes with same source repos content copy not concurrent

Bug 1986356 - pulp3: independent CV publishes with same source repos content copy not concurrent

Summary: pulp3: independent CV publishes with same source repos content copy not concu...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Content Views
Sub Component:
Version:	6.9.0
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	medium
Target Milestone:	6.10.0
Assignee:	satellite6-bugs
QA Contact:	Lai
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1957813
TreeView+	depends on / blocked

Reported:	2021-07-27 10:52 UTC by Peter Vreman
Modified:	2021-11-16 14:13 UTC (History)
CC List:	10 users (show)
Fixed In Version:	tfm-rubygem-katello-4.1.1.28-1
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-11-16 14:12:53 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
verified 8 CVs published at the same time (86.13 KB, image/png) 2021-10-12 15:35 UTC, Lai	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Foreman Issue Tracker	32460	Normal	Closed	improve speed of content view publish with filters	2021-09-15 21:16:18 UTC
Foreman Issue Tracker	33510	Normal	Closed	Composite CV publishing is broken	2021-09-17 19:31:44 UTC
Pulp Redmine	9401	High	CLOSED - CURRENTRELEASE	Backport #9266 "repository 'modify' touches all the content being added one by one" to 3.14.z	2021-09-29 19:06:56 UTC
Red Hat Product Errata	RHSA-2021:4702	None	None	None	2021-11-16 14:12:59 UTC

Description Peter Vreman 2021-07-27 10:52:57 UTC

Description of problem:
If i create 8 ContentViews that contains the same (i guess even 1 same repo is already enough) source repos then the tasks are not parallelized in pulp3.

Looking at the resources and locking code for the tasks it looks like the locking is always done in a write-exclusive way.





Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create 8 pulp workers
2. Sync a few big RHEL7 repos
3. Create 8 CVs with the use ~5 big RHEL7 repos
4. Publish all 8 CVs concurrent

Actual results:
Not all 8 pulp workers are handling the work. it looks like especially the Copy of the content kept in waiting until the previous work is finished

Expected results:
All 8 CVs are published at the same time, almost all 8 pulp workers are busy also during copy of content from the same source repo


Additional info:
Katello has implements in its task locks that have exclude/non-exclusive read/write/link
With Pulp2 the tasks was processes nicely in parallel

Comment 1 Pavel Moravec 2021-07-27 13:42:54 UTC

I can confirm this. My similar reproducer:

- sync a repo (e.g. RH SCL, not a huge one but big enough to see the sequential activity that can be paralelised)
- create several CVs with that repo, e.g. with include filters (just to differentiate the CVs, not required, though); below are hammer commands for it (repo id is 4, customise it if required):

 content-view create --name CV_SCL_filter_no_include_2020-01-01 --organization-id=1 --repository-ids=4 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-01-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-01-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-01-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-01-01'
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-01-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view create --name CV_SCL_filter_no_include_2020-04-01 --organization-id=1 --repository-ids=4 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-04-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-04-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-04-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-04-01'
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-04-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view create --name CV_SCL_filter_no_include_2020-07-01 --organization-id=1 --repository-ids=4 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-07-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-07-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-07-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-07-01'
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-07-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view create --name CV_SCL_filter_no_include_2020-10-01 --organization-id=1 --repository-ids=4 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-10-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-10-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-10-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-10-01'
 content-view filter create --organization-id=1 --content-view=CV_SCL_filter_no_include_2020-10-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=CV_SCL_filter_no_include_2020-01-01 --organization-id=1 --async
 content-view publish --name=CV_SCL_filter_no_include_2020-04-01 --organization-id=1 --async
 content-view publish --name=CV_SCL_filter_no_include_2020-07-01 --organization-id=1 --async
 content-view publish --name=CV_SCL_filter_no_include_2020-10-01 --organization-id=1 --async

Then, monitoring "rq info -u redis://localhost:6379/8" output, I do see that very often just one worker is busy at a time:
grep -e Updated -e busy ./monitor_pulp.sh.log | grep -A1 busy
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:18:48.388602
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:19:03.991474
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:19:19.083449
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:19:33.922867
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:19:48.834893
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:20:03.884128
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:20:18.834495
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:20:33.711916
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:20:48.714693
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:21:03.605449
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:21:18.386906
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:21:33.100790
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:21:47.955466
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:22:02.836547
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:22:17.686315
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:22:32.516278
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:22:47.294468
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:23:02.332730
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:23:17.228638
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:23:32.003726
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:23:46.848843
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:24:01.563018
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:24:16.498382
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:24:31.348888
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:24:46.216439
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:25:01.015165
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:25:15.855349
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:25:30.606393
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:25:45.387846
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
Updated: 2021-07-27 15:26:00.156853
3645.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3645): busy 3645.redhat.com
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:26:15.585264
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:26:30.300150
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:26:44.861166
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:26:59.637141
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:27:14.376347
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:27:29.190430
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:27:43.856932
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:27:58.684808
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
Updated: 2021-07-27 15:28:13.631261
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:28:28.577767
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
Updated: 2021-07-27 15:28:43.353376
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:28:58.005092
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
Updated: 2021-07-27 15:29:12.637948
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:29:27.374615
3653.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3653): busy 3653.redhat.com
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:29:42.115118
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:29:56.847422
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:30:12.337751
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:30:29.791768
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:30:48.218212
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:31:05.191842
3654.redhat.com (pmoravec-sat610-4.satotest.redhat.com 3654): busy 3654.redhat.com
Updated: 2021-07-27 15:31:22.390454

checking postgres data more directly shows, that the concurrent work was for publishing repos or creating them. But all "pulp_rpm.app.tasks.copy.copy_content" type tasks were executed by a single worker.

The latest statement can be verified when expanding dynflow console for the CV publish tasks, and Actions::Pulp3::Repository::MultiCopyContent step in particular. Grepping for:

  name: pulp_rpm.app.tasks.copy.copy_content
  worker: "/pulp/api/v3/workers/9839bce0-15bf-473a-8079-c61632bf8bea/"

I always see just the one worker (the 3654@pmora.. one).


TL;DR: there is really just one type of lock (reservation , in pulp terminology), that acquires the lock for _either_ read or write access. Thus pulp resource manager prevents concurrent read access of a repo.

So I assume an isolated reproducer outside Satellite would be:
- create a pulp repo with a feed, sync it
- create N repos (N>4)
- call pulp_rpm.app.tasks.copy.copy_content from the synced repo to the N repos concurrently
- check if the tasks will be executed concurrently or sequentially

Comment 2 pulp-infra@redhat.com 2021-07-27 14:07:27 UTC

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 3 pulp-infra@redhat.com 2021-07-27 14:07:30 UTC

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2021-07-28 11:06:23 UTC

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 5 Daniel Alley 2021-07-29 20:26:58 UTC

@pmoravec Note that 6.10 has an entirely different tasking system from 6.9 - I see "610" in your address but you are using / monitoring RQ tasking system.  How is your environment set up, and can you reproduce this using the 6.10 (non-RQ-based) tasking system?

Comment 6 Pavel Moravec 2021-07-29 20:32:11 UTC

(In reply to Daniel Alley from comment #5)
> @pmoravec Note that 6.10 has an entirely different tasking system from 6.9 -
> I see "610" in your address but you are using / monitoring RQ tasking
> system.  How is your environment set up, and can you reproduce this using
> the 6.10 (non-RQ-based) tasking system?

I was using snap4 of 6.10, in particular:

satellite-6.10.0-0.1.beta.el7sat.noarch
python3-pulpcore-3.11.2-2.el7pc.noarch
python3-pulp-container-2.5.3-1.el7pc.noarch
python3-pulp-file-1.6.0-1.el7pc.noarch
python3-pulp-ansible-0.7.3-1.el7pc.noarch
python3-pulp-certguard-1.2.0-1.el7pc.noarch
python3-pulp-rpm-3.11.0-1.el7pc.noarch

I wasnt aware of a change from rq to yet something else (btw how to get similar data to "qpid-stat" / "rq", then?). Anyway, I noticed this:

> The latest statement can be verified when expanding dynflow console for the CV publish tasks, and Actions::Pulp3::Repository::MultiCopyContent step in particular. Grepping for:
>
>  name: pulp_rpm.app.tasks.copy.copy_content
>  worker: "/pulp/api/v3/workers/9839bce0-15bf-473a-8079-c61632bf8bea/"
>
> I always see just the one worker (the 3654@pmora.. one).

for each and every copy_content task in the independent CV publishes.

Comment 7 Daniel Alley 2021-07-29 21:44:13 UTC

>I was using snap4 of 6.10

6.10 Snap 9 rebased everything to pulpcore 3.14, and it has a new tasking system. (The old one is still present but is disabled by default).  The reason for this is that we've had unanimous feedback from upstream users that the new tasking system is more performant and more stable - many users that had previously been having a lot of issues with the existing tasking system have had no issues at all with the new system.

So given how different Snap 9+ is from Snap 4 I would strongly recommend using the newer builds for testing. (Even apart from the tasking system there have been a huge number of bugfixes in the past 6 weeks since snap 4)

>(btw how to get similar data to "qpid-stat" / "rq", then?)

Query the status API for online workers and the task API for running tasks. It appears that pulp-cli is included in the newer snaps as well, so "pulp status" and "pulp task list --state=running" will cover that.  But if there is any information that you need which isn't available, let us know and we can figure something out.

Comment 9 Matthias Dellweg 2021-07-30 08:21:08 UTC

(In reply to Daniel Alley from comment #5)
> @pmoravec Note that 6.10 has an entirely different tasking system from 6.9 -
> I see "610" in your address but you are using / monitoring RQ tasking
> system.  How is your environment set up, and can you reproduce this using
> the 6.10 (non-RQ-based) tasking system?

While that is true, we are in the lucky position to know that the use of exclusive locks are the root cause of non-parallelized copy actions. And I know this problem has not yet been faced in the queueless tasking system. For me that is enough theoretical proof to know the issue will persist.

(In reply to Pavel Moravec from comment #6)
[...]
> I wasnt aware of a change from rq to yet something else (btw how to get
> similar data to "qpid-stat" / "rq", then?). Anyway, I noticed this:
> 
> > The latest statement can be verified when expanding dynflow console for the CV publish tasks, and Actions::Pulp3::Repository::MultiCopyContent step in particular. Grepping for:
> >
> >  name: pulp_rpm.app.tasks.copy.copy_content
> >  worker: "/pulp/api/v3/workers/9839bce0-15bf-473a-8079-c61632bf8bea/"
> >
> > I always see just the one worker (the 3654@pmora.. one).
> 
> for each and every copy_content task in the independent CV publishes.

The queueless tasking system removed rq, as well as the resource-manager without a substitute. The workers will however use Postgres advisory locks to synchronize assignment of tasks. Probably the deepest insight can be drawn from a `SELECT * FROM pg_locks;` in a Postgres shell.
However while the tasks will still be waiting on one another, i expect the following two look different:
1. Task will not be assigned to workers before actually starting them (each worker will always have at most one task assigned).
2. Not all tasks waiting for the same resource will be routed to the same worker.

Comment 10 Daniel Alley 2021-07-30 13:26:49 UTC

I filed https://pulp.plan.io/issues/9183 for improved monitoring via the status and worker APIs.

Comment 11 Pavel Moravec 2021-08-06 08:04:55 UTC

(In reply to Daniel Alley from comment #7)
> >I was using snap4 of 6.10
> 
> 6.10 Snap 9 rebased everything to pulpcore 3.14, and it has a new tasking
> system. (The old one is still present but is disabled by default).  The
> reason for this is that we've had unanimous feedback from upstream users
> that the new tasking system is more performant and more stable - many users
> that had previously been having a lot of issues with the existing tasking
> system have had no issues at all with the new system.
> 
> So given how different Snap 9+ is from Snap 4 I would strongly recommend
> using the newer builds for testing. (Even apart from the tasking system
> there have been a huge number of bugfixes in the past 6 weeks since snap 4)
> 
> >(btw how to get similar data to "qpid-stat" / "rq", then?)
> 
> Query the status API for online workers and the task API for running tasks.
> It appears that pulp-cli is included in the newer snaps as well, so "pulp
> status" and "pulp task list --state=running" will cover that.  But if there
> is any information that you need which isn't available, let us know and we
> can figure something out.

I tried that (python3-pulp-rpm-3.14.0-1.el7pc.noarch, snap 3 days old) and I do see purely sequential pulp_rpm.app.tasks.copy.copy_content tasks when I run 2 tests "publish multiple very similar CVs that copy from RHEL7 repo":

pulpcore=# select pulp_id,started_at,finished_at,worker_id,reserved_resources_record from core_task where name = 'pulp_rpm.app.tasks.copy.copy_content' order by started_at;
               pulp_id                |          started_at           |          finished_at          |              worker_id               |                                                
             reserved_resources_record                                                             
--------------------------------------+-------------------------------+-------------------------------+--------------------------------------+------------------------------------------------
---------------------------------------------------------------------------------------------------
 5acba984-9710-49cd-8b60-53d6c83e1198 | 2021-08-05 13:17:27.068845+02 | 2021-08-05 13:17:49.997162+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/5eefc810-c42
d-420c-aebc-ec72634d8de7/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 00da5f1d-bcba-442e-9c4b-f8866c94fac4 | 2021-08-05 13:17:50.106452+02 | 2021-08-05 13:18:20.3903+02   | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/5eefc810-c42
d-420c-aebc-ec72634d8de7/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 f8100aca-a05e-4c8d-abee-0b047788e55a | 2021-08-05 13:18:20.59347+02  | 2021-08-05 13:18:49.050061+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/5eefc810-c42
d-420c-aebc-ec72634d8de7/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 4533b5c3-410c-4d00-85ab-be51d0e84f67 | 2021-08-05 13:18:49.364682+02 | 2021-08-05 13:26:04.880357+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/5eefc810-c42
d-420c-aebc-ec72634d8de7/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 dffb479d-fd02-43b5-a9ed-fbd02038e76b | 2021-08-05 13:26:05.307683+02 | 2021-08-05 13:26:27.952029+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/94e4885e-bc3
0-4bec-958b-562920db72ba/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 c22cdbcc-2d96-4022-b245-94caf12bde31 | 2021-08-05 13:26:28.364119+02 | 2021-08-05 13:27:05.78123+02  | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/94e4885e-bc3
0-4bec-958b-562920db72ba/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 b8f893c5-6ac5-4382-88be-389f1f267dc1 | 2021-08-05 13:27:06.262251+02 | 2021-08-05 13:27:31.745779+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/94e4885e-bc3
0-4bec-958b-562920db72ba/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 e3e645a6-36ca-4547-aa05-828f552a6ddf | 2021-08-05 13:27:32.015279+02 | 2021-08-05 13:38:40.290455+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/94e4885e-bc3
0-4bec-958b-562920db72ba/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 ae309db5-b1c6-4efe-b315-51a1323a77ba | 2021-08-05 13:38:40.563916+02 | 2021-08-05 13:39:03.348511+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/92d0c215-be3
5-4038-a633-18e916be475c/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 cca2c4ab-47f2-4fdd-a2a0-bbafb66ad76d | 2021-08-05 13:39:03.60179+02  | 2021-08-05 13:39:30.067644+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/92d0c215-be3
5-4038-a633-18e916be475c/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 778e6d7f-9524-411c-a584-aacd70158d24 | 2021-08-05 13:39:30.341719+02 | 2021-08-05 13:39:56.234888+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/92d0c215-be3
5-4038-a633-18e916be475c/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 8671fa0b-5e6f-4e44-a037-7d00fb97cf13 | 2021-08-05 13:39:56.470562+02 | 2021-08-05 13:47:43.525421+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/92d0c215-be3
5-4038-a633-18e916be475c/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 488a5c4d-6c0f-4deb-95b0-5cc5433cf66d | 2021-08-05 13:47:43.736198+02 | 2021-08-05 13:48:06.321124+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/8e111767-a90
5-4ccc-a88d-e4994755b1b0/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 43e511f4-d708-4c65-8c2a-8261f44ff770 | 2021-08-05 13:48:06.609826+02 | 2021-08-05 13:48:31.821264+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/8e111767-a90
5-4ccc-a88d-e4994755b1b0/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 befa1b6e-c89b-4c64-8e1e-03626a5e77fc | 2021-08-05 13:48:32.055474+02 | 2021-08-05 13:48:58.077241+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/8e111767-a90
5-4ccc-a88d-e4994755b1b0/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 891f7ce8-0b50-4e53-b936-ffcaae2a85c5 | 2021-08-05 13:48:58.254821+02 | 2021-08-05 13:55:51.315472+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/8e111767-a90
5-4ccc-a88d-e4994755b1b0/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 f493319e-b92c-4a60-90fc-35cd41ab66da | 2021-08-05 13:55:51.561512+02 | 2021-08-05 13:56:14.779442+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/1d0cbea6-dfe
d-4452-9bdc-8ae53ed2119e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 20a5dcb2-a829-4aaa-ab4f-c199195b7ebb | 2021-08-05 13:56:15.065549+02 | 2021-08-05 13:56:54.871561+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/1d0cbea6-dfe
d-4452-9bdc-8ae53ed2119e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 a01ba9be-4067-4e12-a5a4-b1a24d528e95 | 2021-08-05 13:56:55.108737+02 | 2021-08-05 13:57:27.844195+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/1d0cbea6-dfe
d-4452-9bdc-8ae53ed2119e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 4eeb8928-bd84-4233-bb6e-4c0bf65d6d9b | 2021-08-05 13:57:28.035317+02 | 2021-08-05 14:05:13.951424+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/1d0cbea6-dfe
d-4452-9bdc-8ae53ed2119e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 0a2d1ee3-9d09-408d-a6ea-46108e671941 | 2021-08-05 14:05:14.127181+02 | 2021-08-05 14:05:37.820549+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/24a1e2b2-7e5
6-4ad7-9f07-a5555bb4381f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 86f08e48-0b71-4323-85b3-158524c2062d | 2021-08-05 14:05:37.963569+02 | 2021-08-05 14:06:14.175216+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/24a1e2b2-7e5
6-4ad7-9f07-a5555bb4381f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 578b9313-1920-473d-9e62-996fb34eac9b | 2021-08-05 14:06:14.274414+02 | 2021-08-05 14:06:40.733139+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/24a1e2b2-7e5
6-4ad7-9f07-a5555bb4381f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 3abf2c12-5a52-42de-872e-1ed47e71cf0e | 2021-08-05 14:06:40.810432+02 | 2021-08-05 14:13:56.697231+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/24a1e2b2-7e5
6-4ad7-9f07-a5555bb4381f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}

(2nd test)

 3d4316d7-a559-457a-814d-24751290f0a1 | 2021-08-05 15:44:42.715236+02 | 2021-08-05 15:57:40.214462+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/9cd33ddc-9a2
4-4efb-96aa-47bb919287a6/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 9502e0ff-a377-4436-8d18-2b37607ddf1f | 2021-08-05 15:57:40.995368+02 | 2021-08-05 15:59:16.263935+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/9cd33ddc-9a2
4-4efb-96aa-47bb919287a6/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 cd039c52-9949-464c-9633-762ecadc38dc | 2021-08-05 15:59:17.047343+02 | 2021-08-05 16:01:15.771074+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/59256b8d-f58
4-4521-b0b7-41db373cc903/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 b16f7d4b-0815-4cac-b5fa-7923b35dea84 | 2021-08-05 16:01:16.882169+02 | 2021-08-05 16:02:55.125464+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/9cd33ddc-9a2
4-4efb-96aa-47bb919287a6/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 2ac17c35-2716-4627-bc3e-5df158af445b | 2021-08-05 16:02:55.808191+02 | 2021-08-05 16:03:53.963331+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/59256b8d-f58
4-4521-b0b7-41db373cc903/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 2d0ecd0e-497b-4514-92c8-83277e88a8ec | 2021-08-05 16:03:54.281654+02 | 2021-08-05 16:16:19.044757+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/9cd33ddc-9a2
4-4efb-96aa-47bb919287a6/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 af8696e7-9a58-44f6-9942-df9fbe7587ec | 2021-08-05 16:16:20.065688+02 | 2021-08-05 16:17:37.991582+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/59256b8d-f58
4-4521-b0b7-41db373cc903/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 83a70c0b-3f6e-48b6-8d3d-68947023b667 | 2021-08-05 16:17:38.887708+02 | 2021-08-05 16:29:02.470901+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/59256b8d-f58
4-4521-b0b7-41db373cc903/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 bd96ad1f-40b2-4323-b954-7d77c2d36841 | 2021-08-05 16:29:02.839841+02 | 2021-08-05 16:29:26.168354+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/8173fa41-cfe
1-40b9-a5d3-bee41facd849/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 41a2358a-89cf-4c95-b655-7e1e61fa3ead | 2021-08-05 16:29:26.609568+02 | 2021-08-05 16:29:50.19588+02  | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/8173fa41-cfe
1-40b9-a5d3-bee41facd849/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 dc079754-57e8-469e-b260-0c1abf63aef1 | 2021-08-05 16:29:50.587348+02 | 2021-08-05 16:30:33.451146+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/8173fa41-cfe
1-40b9-a5d3-bee41facd849/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 26f49a24-0fda-433f-85a7-9265e70629fd | 2021-08-05 16:30:34.746787+02 | 2021-08-05 16:38:44.963105+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/8173fa41-cfe
1-40b9-a5d3-bee41facd849/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 18f88cf5-93bd-411e-a092-5865cb524763 | 2021-08-05 16:38:45.449139+02 | 2021-08-05 16:39:09.922258+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/d297be19-9b9
e-4c21-bdea-22e352b30cf4/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 8c290449-75d3-4565-b3dd-bed51fd11363 | 2021-08-05 16:39:10.430239+02 | 2021-08-05 16:39:39.845799+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/d297be19-9b9
e-4c21-bdea-22e352b30cf4/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 af9f84e2-175e-49b0-8c8d-293b299dd496 | 2021-08-05 16:39:40.216389+02 | 2021-08-05 16:40:05.646134+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/d297be19-9b9
e-4c21-bdea-22e352b30cf4/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 7678d67a-6b04-47f8-a37e-92e29a855fab | 2021-08-05 16:40:06.006929+02 | 2021-08-05 16:46:55.262114+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/d297be19-9b9
e-4c21-bdea-22e352b30cf4/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 4c76f5b5-8a1b-43c4-8bf2-9f5cc43477a9 | 2021-08-05 16:46:55.698878+02 | 2021-08-05 16:47:18.892304+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/2f94b981-f65
1-4c04-b9be-2f8a7366f28e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 0f9e4342-8668-446f-b52e-7014521636ec | 2021-08-05 16:47:19.320272+02 | 2021-08-05 16:47:59.072242+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/2f94b981-f65
1-4c04-b9be-2f8a7366f28e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 f56c55b1-014b-4bdb-8554-980ca526d88a | 2021-08-05 16:47:59.421917+02 | 2021-08-05 16:48:40.245395+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/2f94b981-f65
1-4c04-b9be-2f8a7366f28e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 6ef64acd-dd3f-4aec-9142-94bc419fd3b6 | 2021-08-05 16:48:40.60293+02  | 2021-08-05 16:55:35.99981+02  | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/2f94b981-f65
1-4c04-b9be-2f8a7366f28e/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 58dc163e-c46f-4770-92bb-e5a3abcb5ed4 | 2021-08-05 16:55:36.424154+02 | 2021-08-05 16:56:01.845191+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/892d32be-163
5-4e21-b92c-51ea81a86017/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 b0baf288-3b71-4681-98f2-6496144350a8 | 2021-08-05 16:56:02.144293+02 | 2021-08-05 16:56:35.910307+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/892d32be-163
5-4e21-b92c-51ea81a86017/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 207f8f07-fcf9-4131-8cca-7fa6c9a4ac8f | 2021-08-05 16:56:36.314782+02 | 2021-08-05 16:57:01.780987+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/892d32be-163
5-4e21-b92c-51ea81a86017/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 24746420-6d7c-4812-9441-ea51f364d89b | 2021-08-05 16:57:02.021393+02 | 2021-08-05 17:04:40.123753+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/892d32be-163
5-4e21-b92c-51ea81a86017/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 2b7da7a1-b54d-4cd1-a944-043683457305 | 2021-08-05 17:04:40.520619+02 | 2021-08-05 17:05:03.072648+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/47914a82-432
f-4e30-b2cb-f0eaa7c1bc3b/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 7f066590-d648-4aaf-8776-e9827996af1d | 2021-08-05 17:05:03.441553+02 | 2021-08-05 17:05:35.679914+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/47914a82-432
f-4e30-b2cb-f0eaa7c1bc3b/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 4b11909c-c494-46cd-a08e-e0ba690c33c3 | 2021-08-05 17:05:36.015679+02 | 2021-08-05 17:06:01.794754+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/47914a82-432
f-4e30-b2cb-f0eaa7c1bc3b/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 62e66d8d-2b3e-459c-aeae-dbd30e743470 | 2021-08-05 17:06:02.131591+02 | 2021-08-05 17:12:56.394558+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/47914a82-432
f-4e30-b2cb-f0eaa7c1bc3b/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 38fee6bf-724b-4fba-9708-9069ee7ca0bf | 2021-08-05 17:12:56.77327+02  | 2021-08-05 17:13:19.541374+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/e2af8eac-849
f-40ab-8a2d-b14ad8733e1f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 d990e2f6-cb06-47cc-81fe-2543f3b161d3 | 2021-08-05 17:13:19.840064+02 | 2021-08-05 17:13:46.943323+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/e2af8eac-849
f-40ab-8a2d-b14ad8733e1f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 002aba47-182b-46e7-b1c3-d0a101cdb156 | 2021-08-05 17:13:47.209305+02 | 2021-08-05 17:14:12.557466+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/e2af8eac-849
f-40ab-8a2d-b14ad8733e1f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 9a4863d8-3443-4de4-b03a-c573fe1649aa | 2021-08-05 17:14:12.825616+02 | 2021-08-05 17:21:16.349318+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/e2af8eac-849
f-40ab-8a2d-b14ad8733e1f/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 c4569308-5629-4f7a-9dc8-24a94366c147 | 2021-08-05 17:21:16.658352+02 | 2021-08-05 17:21:39.508656+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/10bd452f-66e
a-469a-a103-abcd6aa66af5/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 a21b451c-fa04-49e3-ac09-a8aa9b037f18 | 2021-08-05 17:21:39.761456+02 | 2021-08-05 17:22:15.504425+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/10bd452f-66e
a-469a-a103-abcd6aa66af5/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 95345961-844c-4ce0-8590-1f9108af3035 | 2021-08-05 17:22:15.715851+02 | 2021-08-05 17:22:45.566667+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/10bd452f-66e
a-469a-a103-abcd6aa66af5/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 d86de8be-81f3-4d8b-b67e-6b68b40b0165 | 2021-08-05 17:22:45.853312+02 | 2021-08-05 17:29:52.484562+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/10bd452f-66e
a-469a-a103-abcd6aa66af5/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 f0f10772-7855-4fb2-9096-3b66602e228c | 2021-08-05 17:29:52.706503+02 | 2021-08-05 17:30:39.445134+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/d44df829-be6
9-4268-b18a-f49f2a6ea1c2/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 080e94cd-253f-40bb-bd3b-84bfa2430973 | 2021-08-05 17:30:40.17909+02  | 2021-08-05 17:31:36.714718+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/d44df829-be6
9-4268-b18a-f49f2a6ea1c2/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 50bdd54c-9b16-4de8-a785-cb107a4be0f2 | 2021-08-05 17:31:36.865987+02 | 2021-08-05 17:32:27.744151+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/d44df829-be6
9-4268-b18a-f49f2a6ea1c2/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 49621ac3-f797-48ff-a853-353cadcc0315 | 2021-08-05 17:32:27.883321+02 | 2021-08-05 17:39:06.971332+02 | 62efbe4f-c22d-4500-8d0c-2c8962c097cc | {/pulp/api/v3/repositories/rpm/rpm/d44df829-be6
9-4268-b18a-f49f2a6ea1c2/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 d7f3646a-ffc2-44fd-b5eb-5f39245cdee2 | 2021-08-05 17:39:07.179024+02 | 2021-08-05 17:39:28.894285+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/87d548c7-7eb
2-4295-995b-d0e70516ed04/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 486bfc5e-3492-4d45-8898-8b337c49ae83 | 2021-08-05 17:39:29.054119+02 | 2021-08-05 17:40:05.230559+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/87d548c7-7eb
2-4295-995b-d0e70516ed04/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 21750701-b605-4bf3-86ed-623336fa45ee | 2021-08-05 17:40:05.415196+02 | 2021-08-05 17:40:55.977357+02 | 36598f7c-28e3-420a-bbb7-0684abbd931f | {/pulp/api/v3/repositories/rpm/rpm/87d548c7-7eb
2-4295-995b-d0e70516ed04/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 29c807ce-6a94-45a6-b455-df6fb2295fc8 | 2021-08-05 17:40:56.113043+02 | 2021-08-05 17:48:10.623627+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/87d548c7-7eb
2-4295-995b-d0e70516ed04/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 5144e36f-1ed9-41a4-9570-f291ce25a7c1 | 2021-08-05 17:48:10.779199+02 | 2021-08-05 17:48:33.37971+02  | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/b473c560-941
8-4e4a-8fc0-c759ed23e7ca/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 ac2a9782-e347-4515-97e9-dacf4d1e0799 | 2021-08-05 17:48:33.486419+02 | 2021-08-05 17:49:09.219556+02 | 75d17940-07ab-41c0-9e72-209232a0d804 | {/pulp/api/v3/repositories/rpm/rpm/b473c560-941
8-4e4a-8fc0-c759ed23e7ca/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 837dda3d-8146-4be7-a2c8-76bae76edefa | 2021-08-05 17:49:09.356356+02 | 2021-08-05 17:49:59.454341+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/b473c560-941
8-4e4a-8fc0-c759ed23e7ca/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}
 8640ac16-f4e4-4886-bbe1-6ae6d1fc8c42 | 2021-08-05 17:49:59.560706+02 | 2021-08-05 17:57:20.243624+02 | 53738c13-d527-4171-8640-701edbdab719 | {/pulp/api/v3/repositories/rpm/rpm/b473c560-941
8-4e4a-8fc0-c759ed23e7ca/,/pulp/api/v3/repositories/rpm/rpm/ed6d41b2-14d0-41f3-8566-3080562d69c9/}


Although different workers can be assigned to individual tasks, the tasks are still executed in a sequence.


Particular reproducer: pass below text to hammer shell to create & publish CVs with RHEL7 repo (repository-id 1, update if needed) and with some include filter (just to differentiate the CV content, imho redundant for the test case).

 content-view create --name CV_RHEL7_no_include_2020-02-01 --organization-id=1 --repository-ids=1 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-02-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-02-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-02-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-02-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-02-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=CV_RHEL7_no_include_2020-02-01 --organization-id=1 --async
 content-view create --name CV_RHEL7_no_include_2020-04-01 --organization-id=1 --repository-ids=1 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-04-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-04-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-04-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-04-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-04-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=CV_RHEL7_no_include_2020-04-01 --organization-id=1 --async
 content-view create --name CV_RHEL7_no_include_2020-06-01 --organization-id=1 --repository-ids=1 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-06-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-06-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-06-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-06-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-06-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=CV_RHEL7_no_include_2020-06-01 --organization-id=1 --async
 content-view create --name CV_RHEL7_no_include_2020-08-01 --organization-id=1 --repository-ids=1 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-08-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-08-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-08-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-08-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-08-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=CV_RHEL7_no_include_2020-08-01 --organization-id=1 --async
 content-view create --name CV_RHEL7_no_include_2020-10-01 --organization-id=1 --repository-ids=1 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-10-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-10-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-10-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-10-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-10-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=CV_RHEL7_no_include_2020-10-01 --organization-id=1 --async
 content-view create --name CV_RHEL7_no_include_2020-12-01 --organization-id=1 --repository-ids=1 --solve-dependencies=no
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-12-01 --name=include_base --inclusion=true --original-packages=true --type=rpm
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-12-01 --name=include_errata --inclusion=true --type=erratum
 content-view filter rule create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-12-01 --content-view-filter=include_errata --date-type='updated' --end-date='2020-12-01'
 content-view filter create --organization-id=1 --content-view=CV_RHEL7_no_include_2020-12-01 --name=include_modules --inclusion=true --original-module-streams=true --type=modulemd
 content-view publish --name=CV_RHEL7_no_include_2020-12-01 --organization-id=1 --async

Comment 12 pulp-infra@redhat.com 2021-08-19 10:08:51 UTC

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2021-08-19 11:06:47 UTC

All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 14 pulp-infra@redhat.com 2021-08-19 12:12:12 UTC

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2021-08-19 12:12:14 UTC

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 16 pulp-infra@redhat.com 2021-08-26 13:15:38 UTC

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 17 pulp-infra@redhat.com 2021-08-26 16:11:57 UTC

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 18 pulp-infra@redhat.com 2021-08-26 18:09:26 UTC

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 19 pulp-infra@redhat.com 2021-08-26 18:09:35 UTC

All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 20 pulp-infra@redhat.com 2021-08-27 16:11:06 UTC

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 22 Tanya Tereshchenko 2021-09-02 21:03:39 UTC

Repurposing this BZ to be solved only for the standard CV publish with no dependency solving enabled.
Moving it to be solved on Katello side.

The similar issue but with the dependency solving is tracked by BZ#2000769.

Comment 23 Justin Sherrill 2021-09-02 21:05:28 UTC

Connecting redmine issue https://projects.theforeman.org/issues/32460 from this bug

Comment 24 Bryan Kearney 2021-09-04 00:05:27 UTC

Upstream bug assigned to iballou

Comment 25 Bryan Kearney 2021-09-04 00:05:29 UTC

Upstream bug assigned to iballou

Comment 28 Robin Chan 2021-09-21 21:35:53 UTC

@dalley and I correct in understanding that this BZ should point to https://pulp.plan.io/issues/9401 instead of https://pulp.plan.io/issues/9266?

Comment 29 Daniel Alley 2021-09-21 21:41:09 UTC

Yes, I believe so.

Comment 30 pulp-infra@redhat.com 2021-09-22 18:08:56 UTC

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 31 pulp-infra@redhat.com 2021-09-22 18:08:57 UTC

The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 32 pulp-infra@redhat.com 2021-09-23 19:06:33 UTC

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 36 pulp-infra@redhat.com 2021-09-29 02:17:53 UTC

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 37 pulp-infra@redhat.com 2021-09-29 19:06:57 UTC

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 42 pulp-infra@redhat.com 2021-10-11 20:08:26 UTC

Requesting needsinfo from upstream developer dalley because the 'FailedQA' flag is set.

Comment 44 Lai 2021-10-12 15:33:53 UTC

Steps to Retest:

1. Create 8 pulp workers
2. Sync a few big RHEL7 repos
3. Create 8 CVs with the use ~5 big RHEL7 repos
4. Publish all 8 CVs concurrent

Expected results:
All 8 CVs should be published at relatively the same time


Actual:
All 8 CVs are published at relatively the same time.


Verified on 6.10_22 with tfm-rubygem-katello-4.1.1.33-1.el7sat.noarch

Comment 45 Lai 2021-10-12 15:35:06 UTC

Created attachment 1832263 [details]
verified 8 CVs published at the same time

Comment 48 errata-xmlrpc 2021-11-16 14:12:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702

Note You need to log in before you can comment on or make changes to this bug.