Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2122344 - Run pulpcore-manager trim-rpm-changelogs during upgrades to 6.12
Summary: Run pulpcore-manager trim-rpm-changelogs during upgrades to 6.12
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Satellite Maintain
Version: 6.12.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: 6.12.2
Assignee: Evgeni Golov
QA Contact: Griffin Sullivan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-29 20:12 UTC by Daniel Alley
Modified: 2023-10-30 11:42 UTC (History)
14 users (show)

Fixed In Version: rubygem-foreman_maintain-1.1.9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-02-08 14:35:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 35678 0 High Closed Run pulpcore-manager trim-rpm-changelogs during upgrades to pulp_rpm 3.17.5+ 2022-11-01 15:04:24 UTC
Red Hat Issue Tracker SAT-14082 0 None None None 2022-11-28 18:02:37 UTC
Red Hat Issue Tracker SAT-14623 0 None None None 2023-01-06 13:17:21 UTC
Red Hat Product Errata RHBA-2023:0668 0 None None None 2023-02-08 14:35:17 UTC

Description Daniel Alley 2022-08-29 20:12:16 UTC
In order to counter https://issues.redhat.com/browse/RHELDST-11565, in Satellite 6.10 we introduced a changelog limiting feature which restricted the number of changelogs saved into Satellite to the latest 10 (by default).

However this only applies to brand new installations. Any packages synced from the CDN prior to the application of that patch, or migrated from Pulp 2, will not be re-saved with this restriction in place.  In effect this means that the package metadata saved for these old packages is larger than it needs to be by significant margins.

The impacts:

* more memory consumption during import / export operations [0]
* longer publishes [1]
* significant amounts of unnecessary disk space utilization [1]

We added a script which would clean up this unnecessary data, but nothing triggers it to be executed. It's something that only ever needs to be run once, so having "foreman-maintain upgrade" run it would result in waste.

Therefore we ought to document, as a one-time operation, running the following command (I believe this is correct):

sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' /usr/bin/pulpcore-manager rpm-trim-changelogs

We can do this as a manual upgrade step for 6.12 but it can be run on any version 6.11+ (and on 6.10 with a slight tweak).  Therefore we can also suggest this to any users who are encountering problems with these items as well.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=2061224#c28
[1] https://github.com/pulp/pulp_rpm/pull/2333#issuecomment-1006024014

Comment 2 Daniel Alley 2022-08-31 14:24:44 UTC
It can be done at any time, even on 6.11, but for clarity's sake it would be best to recommend it during the 6.12 upgrade.  Performing it during an upgrade downtime event would also prevent interference with ongoing publish operations (as there shouldn't be any).


It is not strictly required to run and it is safe to run more than once by accident, so no concerns there.  But it should significantly reduce the ongoing disk space requirements as well as improve overall performance.

Comment 5 Evgeni Golov 2022-10-13 08:27:22 UTC
(In reply to Daniel Alley from comment #0)

> We added a script which would clean up this unnecessary data, but nothing
> triggers it to be executed. It's something that only ever needs to be run
> once, so having "foreman-maintain upgrade" run it would result in waste.

We can make foreman-maintain only call it "once" (well, twice): during/after 6.10→6.11 and during/after 6.11→6.12, which should ensure that all users got it executed at least once and then forget about it again.

This is IMHO much better then documenting an addition step which users will for sure forget, we will have to write KCS for etc etc.

Comment 6 Daniel Alley 2022-10-13 14:32:11 UTC
If that's possible it may be a good idea, I was under the impression that it would have to run every time, including during minor version upgrades.

Comment 7 Evgeni Golov 2022-10-13 15:28:13 UTC
Nah, we have separate execution plans for each X.Y→X.(Y+1) and for each "to latest X.Y.z" so we have full control when things are executed.

Comment 8 Daniel Alley 2022-10-13 18:01:01 UTC
For what it's worth, as of 6.12 (and 6.11 too if 6.11.4 were to pick up the patch) the changelogs of existing packages can also be slowly cleaned up over time as repositories are re-synced.  So even if there were no manual or automatic installer steps, the situation will improve itself.  That's an option too.

If it doesn't get picked up for the last 6.11 release then we might want to have the KCS anyway.

Comment 9 Daniel Alley 2022-10-17 21:16:13 UTC
@egolov >re: "Nah, we have separate execution plans for each X.Y→X.(Y+1) and for each "to latest X.Y.z" so we have full control when things are executed."

Just double checking, if a user upgraded directly to 6.12.3 or something, that wouldn't skip the step, would it?  Or is that not a thing that can happen.

Also is it not too late for an installer change for 6.12?

Comment 10 Evgeni Golov 2022-10-18 11:20:14 UTC
(In reply to Daniel Alley from comment #9)
> @egolov >re: "Nah, we have separate execution plans for each
> X.Y→X.(Y+1) and for each "to latest X.Y.z" so we have full control when
> things are executed."
> 
> Just double checking, if a user upgraded directly to 6.12.3 or something,
> that wouldn't skip the step, would it?  Or is that not a thing that can
> happen.

If the user is coming from a 6.11.whatever, this step would be executed, yes.

> Also is it not too late for an installer change for 6.12?

For GA? Probably.

Comment 11 Evgeni Golov 2022-10-18 11:51:43 UTC
Here's a patch that should do what you wanted: https://github.com/evgeni/foreman_maintain/commit/pulpcore-trim

Comment 12 Daniel Alley 2022-10-18 15:34:34 UTC
@Evgeni So the question now is whether it's worth putting in the installer at this point if it can't go into 6.12.0

Rather than tweaking that at this point, perhaps it is enough to just let it be resolved over time as they sync their repositories on 6.12, and then decide whether or not it is worth having the 6.13 perform the same fix for whatever hasn't already been cleaned up.

Comment 13 Evgeni Golov 2022-10-18 16:53:24 UTC
not everyone upgrades on day 1, so having this as part of the "to 6.12" path for people who go once 6.12.1 is out sounds like a good thing to me, but I'll leave the decision to @ehelms 

(also, it's foreman maintain, not installer, but that's an implementation detail)

Comment 14 Eric Helms 2022-10-25 12:44:39 UTC
We are moving this to foreman-maintain component to automate it as part of the 6.12 upgrade process, we will consider potentially including as part of the z-stream upgrade process.

@dalley -- do you know what the rough runtime is for this command?

Comment 15 Daniel Alley 2022-10-25 16:31:01 UTC
Do you mean "added to the z-stream upgrade process" or "added to the 6.12 upgrade process in a z-stream"?  If it was part of a z-stream upgrade then would it only run once or would it be triggered on every z-stream upgrade?

The rough runtime depends entirely on the # of RPMs in the installation, and possibly how fast their database access is.  I think we'd probably want to test it out on a dogfood box to get an approximation since my tests on a development system aren't very comparable.

Comment 16 Evgeni Golov 2022-10-26 09:25:09 UTC
Created redmine issue https://projects.theforeman.org/issues/35678 from this bug

Comment 17 Eric Helms 2022-10-26 15:49:12 UTC
Is there any indicator that can be used to determine *if* trim changelogs needs to run or has been previously run?

Comment 18 Daniel Alley 2022-10-26 18:17:02 UTC
Not really, no. And to be clear, if it's not run or if it's run multiple times, there won't be correctness issues or anything like that. This is merely cleaning up a bunch of superfluous data which is indirectly causing memory issues, using up disk space and slowing things down, which was contributing to a few other BZs.

But as mentioned in comment #8 6.12 and forward can progressively clean that up over time (it didn't make it into 6.11.4 though), so I don't know how much energy it's worth spending to make sure we do a big cleanup up-front during the upgrade.  It would be nice to know everything is consistent and cleaned up at a particular point, but not strictly *necessary*.

Comment 19 Eric Helms 2022-10-27 17:12:58 UTC
We have opted to codify this as part of the upgrade process -- given it can save system resources for users it seems best to just do it for them during upgrade rather than rely upon them to see documentation. It is being included as part of the 6.12 and 6.12.z upgrade paths.

Comment 20 Bryan Kearney 2022-10-28 12:04:14 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35678 has been resolved.

Comment 28 Griffin Sullivan 2023-01-25 21:12:51 UTC
Verified on Satellite 6.12.2 snap 1

pulpcore-manager rpm-trim-changelogs gets run during 6.11 to 6.12 upgrade.

Steps to reproduce:

1) Get a 6.11 satellite with some content

2) Run upgrade to 6.12

3) Check logs for rpm-trim-changelogs messages

Results:

INFO -- : --- Execution step 'Trim RPM changelogs in the pulpcore db' [pulpcore-trim-rpm-changelogs] started ---
DEBUG -- : Running command sudo PULP_SETTINGS=/etc/pulp/settings.py DJANGO_SETTINGS_MODULE=pulpcore.app.settings pulpcore-manager rpm-trim-changelogs with stdin nil


You also get some output about trimming changelogs in the upgrade output

Comment 33 errata-xmlrpc 2023-02-08 14:35:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.12.2 Async Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:0668


Note You need to log in before you can comment on or make changes to this bug.