Bug 1309673 - [RFE] make pulp regenerate applicability calculation incremental
[RFE] make pulp regenerate applicability calculation incremental
Status: NEW
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
6.1.6
All All
medium Severity high (vote)
: Unspecified
: --
Assigned To: satellite6-bugs
Katello QA List
: FutureFeature, Improvement, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-18 07:26 EST by Pavel Moravec
Modified: 2017-07-26 15:37 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Pulp Redmine 2125 Normal NEW make pulp regenerate applicability calculation incremental 2016-11-21 13:52 EST

  None (edit)
Description Pavel Moravec 2016-02-18 07:26:44 EST
Description of problem:
Background: regenerate applicability (reg.app.) pulp task recalculates what errata can be applied to what consumer / Content Host. The calculation is done against _profiles_ of consumers instead of individual consumers, where a profile is unique representation of a set of consumers with the same packages installed.

Technically, reg.app. task traverses repo_profile_applicability mongo collection and updates its rows (and triggers updates of consumer_unit_profiles, I expect).

I have noticed that repo_profile_applicability often contains much more profiles than consumer_unit_profiles. That causes reg.app.task to apply its calculation to redundant / orphaned profiles not associated with any consumer. This causes reg.app. task running for _hours_ on a pulp server after a syncing a repo with 1500 consumers and hundreds of RPMs - that activity should take minutes, at most.

When "repo_profile_applicability row leak" occurs? I have one scenario but expect to be more:
0) export mongo collections repo_profile_applicability and consumer_unit_profiles
1) register a Content Host / pulp consumer
2) attach some subscription and enable some repo (never used before by a consumer, ideally)
3) export the mongo collections again
4) unregister the Content Host / pulp consumer
5) export the collections once more

mongo collections in 0 and 5 should be the same. Esp. grep in pretty output for unique profile_hash-es. At 5, some profile_hash for the just deleted consumer will remain in repo_profile_applicability collection.

Repeat the above with different set of installed packages, and you get another orphaned entry.


Version-Release number of selected component (if applicable):
Sat 6.1.6 / pulp-server-2.6.0.19-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. prepare some mongo queries:
echo -e "DBQuery.shellBatchSize = 100000\ndb.repo_profile_applicability.find().pretty()" > repo_profile_applicability.pretty.js
echo -e "DBQuery.shellBatchSize = 100000\ndb.consumer_unit_profiles.find().pretty()" > consumer_unit_profiles.pretty.js

2. Run them:
mongo pulp_database < consumer_unit_profiles.pretty.js > consumer_unit_profiles.pretty.txt.1
mongo pulp_database < repo_profile_applicability.pretty.js > repo_profile_applicability.pretty.txt.1

3. Register a Content Host / pulp consumer, attach some subscription, enable some repo and install whatever RPM from the repo

4. Run the queries again:
mongo pulp_database < consumer_unit_profiles.pretty.js > consumer_unit_profiles.pretty.txt.2
mongo pulp_database < repo_profile_applicability.pretty.js > repo_profile_applicability.pretty.txt.2

5. unregister the Content Host / pulp consumer

6. Run the queries yet again:
mongo pulp_database < consumer_unit_profiles.pretty.js > consumer_unit_profiles.pretty.txt.3
mongo pulp_database < repo_profile_applicability.pretty.js > repo_profile_applicability.pretty.txt.3

7. Compare unique profile_hash-es in the mongo collections gathered:
for file in consumer_unit_profiles.pretty.txt.* repo_profile_applicability.pretty.txt.*; do
  echo $file $(grep profile_hash $file | sort -u | wc -l)
done


Actual results:
repo_profile_applicability.pretty.txt.3 having more unique profile_hash-es than repo_profile_applicability.pretty.txt.1 (in fact .3 has the same number as .2)
consumer_unit_profiles.pretty.txt.* has expected numbers (.1 same as .3, .2 has some extra)


Expected results:
repo_profile_applicability.pretty.txt.3 to have same number of unique profile_hash-es as repo_profile_applicability.pretty.txt.1


Additional info:
This is just one scenario. I bet there are more (like disable some repo on the consumer and run subscription-manager refresh, or so). Please check what other scenarios can lead to the same oprhaned profiles in repo_profile_applicability.

6.1.z asked since having a customer largely impacted. If providing a cleanup script to safely remove orphaned entries from the collection, 6.1.z flag can be dropped.
Comment 1 Pavel Moravec 2016-02-18 07:35:42 EST
Dennis,
could you please provide a workaround in cleaning the orphaned entries / profiles?

I expect it to be:
- list profile-hash that are in repo_profile_applicability but not in consumer_unit_profiles
- delete from repo_profile_applicability any row having profile-hash from that list

Please verify if I am right and prepare some script / mongo query to do so (I still feel I am mongo beginner).
Comment 3 Tanya Tereshchenko 2016-02-19 10:32:22 EST
Pavel,
I've investigated this issue and it looks like applicability is recalculated only for the existing profiles. You can see the code that ensures this behaviour here, there is also a long comment about that:
https://github.com/pulp/pulp/blob/2.6-release/server/pulp/server/managers/consumer/applicability.py#L143-L151
Comment 4 Pavel Moravec 2016-02-20 08:30:37 EST
(In reply to ttereshc from comment #3)
> Pavel,
> I've investigated this issue and it looks like applicability is recalculated
> only for the existing profiles. You can see the code that ensures this
> behaviour here, there is also a long comment about that:
> https://github.com/pulp/pulp/blob/2.6-release/server/pulp/server/managers/
> consumer/applicability.py#L143-L151

Thanks for having a look. Adding some debug, I confirm the regenerate_applicability method is called only for profiles with a consumer. I will check with the customer behind what happens during the regenerate_applicability_for_repos method.

Re https://github.com/pulp/pulp/blob/2.6-release/server/pulp/server/managers/consumer/applicability.py#L149-#L150 : is there a way how to trigger the monthly task manually? (pulp-admin does not sound to have a way to execute a task) I can manually hack the code in calling RepoProfileApplicability.objects.remove_orphans() somewhere, but that's not ideal, I guess.
Comment 5 Tanya Tereshchenko 2016-02-23 10:08:52 EST
(In reply to Pavel Moravec from comment #4)
> (In reply to ttereshc from comment #3)
> > Pavel,
> > I've investigated this issue and it looks like applicability is recalculated
> > only for the existing profiles. You can see the code that ensures this
> > behaviour here, there is also a long comment about that:
> > https://github.com/pulp/pulp/blob/2.6-release/server/pulp/server/managers/
> > consumer/applicability.py#L143-L151
> 
> Thanks for having a look. Adding some debug, I confirm the
> regenerate_applicability method is called only for profiles with a consumer.
> I will check with the customer behind what happens during the
> regenerate_applicability_for_repos method.
> 
> Re
> https://github.com/pulp/pulp/blob/2.6-release/server/pulp/server/managers/
> consumer/applicability.py#L149-#L150 : is there a way how to trigger the
> monthly task manually? (pulp-admin does not sound to have a way to execute a
> task) I can manually hack the code in calling
> RepoProfileApplicability.objects.remove_orphans() somewhere, but that's not
> ideal, I guess.

There is no good way to remove these records, mostly because you do not need to. There is no harm to have them temporarily in db.
Comment 6 Michael Hrivnak 2016-03-03 15:42:32 EST
It looks like this is working as expected, so I suggest closing as not a bug.
Comment 7 Pavel Moravec 2016-03-16 17:19:58 EDT
An idea of substantial improvement: calculate the applicability incrementally

Assume pulp has already synced a big repo with thousands of RPMs and hundreds of erratas. These have been already calculated in regenerate applicability. Now, syncing a new content from an importer brings say tens RPMs and few errata. Can't be the reg.app. task performed _only_ to the new units? Currently whole bunch of all units within the repo is recalculated completely from scratch.
Comment 8 Brian Bouterse 2016-03-23 11:04:24 EDT
Incremental applicability calculations would be great, but Pulp would need versioned repositories for that to be possible. Pulp doesn't have versioned repositories that today. To get incremental applicability on upstream's radar, consider filing an issue in the upstream Pulp issue tracker for that feature request. It's at pulp.plan.io.

Regarding Comment 6, the claim is that it is working as expected. Can this issue either be closed or have more info put on it that shows it's not working? I really value the input you've given so far, but I'm not sure how to proceed without more feedback from you.
Comment 9 Pavel Moravec 2016-03-23 11:22:42 EDT
Despite I dont think regenerate applicability task taking hours for one repo is expected performance (like it was said to me), let deal with this BZ as an improvement to make the calculation incremental.

I request that for Satellite6 that suffers so many performance issues that qualify it as a valid request. I don't think I have to file the same in pulp upstream - that's rather engineering work, not support guy work (yes, I *can* do that, but that does not mean it's part of my job, still).

> but Pulp would need versioned repositories for that to be possible.
Why it can't use the info I see in Satellite task details "5 new packages downloaded" (I guess same info is available for any unit)? My (limited) understanding of repositories is that "repository versions" are individual *-updateinfo.xml.gz files that are downloaded - just the info from newly downloaded files can be used for the incremental reg.app. calculation (I guess same info can be found for other unit types).
Comment 10 Brian Bouterse 2016-03-23 11:36:39 EDT
Thanks for switching to be an improvement.

Knowing which units have been added since the last sync time would be possible without versioned repos, but knowing which units have been deleted from a repo is not. Without that second part, we can't do incremental calculation and guarantee the profile is correct.
Comment 11 Bryan Kearney 2016-07-26 11:25:24 EDT
Moving 6.2 bugs out to sat-backlog.
Comment 12 Bryan Kearney 2016-07-26 11:35:44 EDT
Moving 6.2 bugs out to sat-backlog.
Comment 19 Brian Bouterse 2016-08-03 10:28:42 EDT
Thanks Pavel and Brad. With that kind of timeline in mind I'm glad it's filed in upstream.
Comment 20 pulp-infra@redhat.com 2016-11-21 13:52:08 EST
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 21 pulp-infra@redhat.com 2016-11-21 13:52:12 EST
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Note You need to log in before you can comment on or make changes to this bug.