Red Hat Bugzilla – Bug 1309673
[RFE] make pulp regenerate applicability calculation incremental
Last modified: 2017-07-26 15:37:30 EDT
Description of problem:
Background: regenerate applicability (reg.app.) pulp task recalculates what errata can be applied to what consumer / Content Host. The calculation is done against _profiles_ of consumers instead of individual consumers, where a profile is unique representation of a set of consumers with the same packages installed.
Technically, reg.app. task traverses repo_profile_applicability mongo collection and updates its rows (and triggers updates of consumer_unit_profiles, I expect).
I have noticed that repo_profile_applicability often contains much more profiles than consumer_unit_profiles. That causes reg.app.task to apply its calculation to redundant / orphaned profiles not associated with any consumer. This causes reg.app. task running for _hours_ on a pulp server after a syncing a repo with 1500 consumers and hundreds of RPMs - that activity should take minutes, at most.
When "repo_profile_applicability row leak" occurs? I have one scenario but expect to be more:
0) export mongo collections repo_profile_applicability and consumer_unit_profiles
1) register a Content Host / pulp consumer
2) attach some subscription and enable some repo (never used before by a consumer, ideally)
3) export the mongo collections again
4) unregister the Content Host / pulp consumer
5) export the collections once more
mongo collections in 0 and 5 should be the same. Esp. grep in pretty output for unique profile_hash-es. At 5, some profile_hash for the just deleted consumer will remain in repo_profile_applicability collection.
Repeat the above with different set of installed packages, and you get another orphaned entry.
Version-Release number of selected component (if applicable):
Sat 6.1.6 / pulp-server-220.127.116.11-1.el7sat.noarch
Steps to Reproduce:
1. prepare some mongo queries:
echo -e "DBQuery.shellBatchSize = 100000\ndb.repo_profile_applicability.find().pretty()" > repo_profile_applicability.pretty.js
echo -e "DBQuery.shellBatchSize = 100000\ndb.consumer_unit_profiles.find().pretty()" > consumer_unit_profiles.pretty.js
2. Run them:
mongo pulp_database < consumer_unit_profiles.pretty.js > consumer_unit_profiles.pretty.txt.1
mongo pulp_database < repo_profile_applicability.pretty.js > repo_profile_applicability.pretty.txt.1
3. Register a Content Host / pulp consumer, attach some subscription, enable some repo and install whatever RPM from the repo
4. Run the queries again:
mongo pulp_database < consumer_unit_profiles.pretty.js > consumer_unit_profiles.pretty.txt.2
mongo pulp_database < repo_profile_applicability.pretty.js > repo_profile_applicability.pretty.txt.2
5. unregister the Content Host / pulp consumer
6. Run the queries yet again:
mongo pulp_database < consumer_unit_profiles.pretty.js > consumer_unit_profiles.pretty.txt.3
mongo pulp_database < repo_profile_applicability.pretty.js > repo_profile_applicability.pretty.txt.3
7. Compare unique profile_hash-es in the mongo collections gathered:
for file in consumer_unit_profiles.pretty.txt.* repo_profile_applicability.pretty.txt.*; do
echo $file $(grep profile_hash $file | sort -u | wc -l)
repo_profile_applicability.pretty.txt.3 having more unique profile_hash-es than repo_profile_applicability.pretty.txt.1 (in fact .3 has the same number as .2)
consumer_unit_profiles.pretty.txt.* has expected numbers (.1 same as .3, .2 has some extra)
repo_profile_applicability.pretty.txt.3 to have same number of unique profile_hash-es as repo_profile_applicability.pretty.txt.1
This is just one scenario. I bet there are more (like disable some repo on the consumer and run subscription-manager refresh, or so). Please check what other scenarios can lead to the same oprhaned profiles in repo_profile_applicability.
6.1.z asked since having a customer largely impacted. If providing a cleanup script to safely remove orphaned entries from the collection, 6.1.z flag can be dropped.
could you please provide a workaround in cleaning the orphaned entries / profiles?
I expect it to be:
- list profile-hash that are in repo_profile_applicability but not in consumer_unit_profiles
- delete from repo_profile_applicability any row having profile-hash from that list
Please verify if I am right and prepare some script / mongo query to do so (I still feel I am mongo beginner).
I've investigated this issue and it looks like applicability is recalculated only for the existing profiles. You can see the code that ensures this behaviour here, there is also a long comment about that:
(In reply to ttereshc from comment #3)
> I've investigated this issue and it looks like applicability is recalculated
> only for the existing profiles. You can see the code that ensures this
> behaviour here, there is also a long comment about that:
Thanks for having a look. Adding some debug, I confirm the regenerate_applicability method is called only for profiles with a consumer. I will check with the customer behind what happens during the regenerate_applicability_for_repos method.
Re https://github.com/pulp/pulp/blob/2.6-release/server/pulp/server/managers/consumer/applicability.py#L149-#L150 : is there a way how to trigger the monthly task manually? (pulp-admin does not sound to have a way to execute a task) I can manually hack the code in calling RepoProfileApplicability.objects.remove_orphans() somewhere, but that's not ideal, I guess.
(In reply to Pavel Moravec from comment #4)
> (In reply to ttereshc from comment #3)
> > Pavel,
> > I've investigated this issue and it looks like applicability is recalculated
> > only for the existing profiles. You can see the code that ensures this
> > behaviour here, there is also a long comment about that:
> > https://github.com/pulp/pulp/blob/2.6-release/server/pulp/server/managers/
> > consumer/applicability.py#L143-L151
> Thanks for having a look. Adding some debug, I confirm the
> regenerate_applicability method is called only for profiles with a consumer.
> I will check with the customer behind what happens during the
> regenerate_applicability_for_repos method.
> consumer/applicability.py#L149-#L150 : is there a way how to trigger the
> monthly task manually? (pulp-admin does not sound to have a way to execute a
> task) I can manually hack the code in calling
> RepoProfileApplicability.objects.remove_orphans() somewhere, but that's not
> ideal, I guess.
There is no good way to remove these records, mostly because you do not need to. There is no harm to have them temporarily in db.
It looks like this is working as expected, so I suggest closing as not a bug.
An idea of substantial improvement: calculate the applicability incrementally
Assume pulp has already synced a big repo with thousands of RPMs and hundreds of erratas. These have been already calculated in regenerate applicability. Now, syncing a new content from an importer brings say tens RPMs and few errata. Can't be the reg.app. task performed _only_ to the new units? Currently whole bunch of all units within the repo is recalculated completely from scratch.
Incremental applicability calculations would be great, but Pulp would need versioned repositories for that to be possible. Pulp doesn't have versioned repositories that today. To get incremental applicability on upstream's radar, consider filing an issue in the upstream Pulp issue tracker for that feature request. It's at pulp.plan.io.
Regarding Comment 6, the claim is that it is working as expected. Can this issue either be closed or have more info put on it that shows it's not working? I really value the input you've given so far, but I'm not sure how to proceed without more feedback from you.
Despite I dont think regenerate applicability task taking hours for one repo is expected performance (like it was said to me), let deal with this BZ as an improvement to make the calculation incremental.
I request that for Satellite6 that suffers so many performance issues that qualify it as a valid request. I don't think I have to file the same in pulp upstream - that's rather engineering work, not support guy work (yes, I *can* do that, but that does not mean it's part of my job, still).
> but Pulp would need versioned repositories for that to be possible.
Why it can't use the info I see in Satellite task details "5 new packages downloaded" (I guess same info is available for any unit)? My (limited) understanding of repositories is that "repository versions" are individual *-updateinfo.xml.gz files that are downloaded - just the info from newly downloaded files can be used for the incremental reg.app. calculation (I guess same info can be found for other unit types).
Thanks for switching to be an improvement.
Knowing which units have been added since the last sync time would be possible without versioned repos, but knowing which units have been deleted from a repo is not. Without that second part, we can't do incremental calculation and guarantee the profile is correct.
Moving 6.2 bugs out to sat-backlog.
Thanks Pavel and Brad. With that kind of timeline in mind I'm glad it's filed in upstream.
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.