Bug 2151657

Regarding 

>>>>>>
Steps to Reproduce:
1. Create a custom repository
2. Rename the first package as 'fubar.rpm'
3. Upload the first package
4. Rename the second package as 'fubar.rpm'
5. Upload the second package
6. Rename the third package as 'fubar.rpm'
7. Upload the third package
8. Try to download the first package
9. Try to download the second package
10. Try to download the third package

Actual results:
See 3 packages but only allow to download the third package

Expected results:
See 3 packages and download all of them
>>>>>>

This isn't possible, you cannot have 3 different files sharing the same path name and download all of them.  There is no way to do that regardless of whether you're talking about files or URLs, without changing one of the paths or letting one of them "win".

So I would say that this issue *as written* is invalid. That doesn't mean that the UX cannot be improved, but we need to figure out some way to improve the UX without breaking the laws of the universe :)

In the interim I would recommend not uploading all packages with the name "fubar.rpm" or any pattern similar to this. RPM file names should follow the pattern $name-$version-$release.$arch.rpm

Unfortunately ther is a fundamental contradiction between the needs of users who want Satellite to respect the filename they provided so that they know what to expect from the URLs and can provide the exact filename to yum/dnf, and users who provide nonsensical filenames and want Satellite to do something to work around that, and it's pretty challenging to make everyone happy in that respect.

Here are two semi-related BZs to look at: https://bugzilla.redhat.com/show_bug.cgi?id=2161993, https://bugzilla.redhat.com/show_bug.cgi?id=2162591

If there is an issue which neither of those covers and which makes sense to address, then please file a new BZ.

Yum repositories are only supposed to contain RPMs named $name-$version-$release.$arch.rpm
Unfortunately, some vendors that provide RPMs do not correctly name their RPM files.  For example, IBM provides RPMs for Tivoli, but does not include the version/release in the filename: TIVsm-API64.x86_64.rpm

In previous versions of Satellite (Prior to the switch from Pulp 2 to Pulp 3), Satellite would automatically rename an uploaded RPM file as needed (by extracting the name/version/release/arch from the RPM itself).
In Satellite 6.11, RPMs are not renamed automatically by Satellite, and uploading multiple RPMs with the same filename appears to work at upload time, but then breaks things when you subsequently try to install those files with `yum`.

I consider this to be a bug in Satellite.  At the very least, Satellite should prevent uploads with improper filenames and provide the user with instructions/documentation for renaming the file appropriately.  Ideally, the automatic renaming behavior that existed with Pulp 2 should be restored so that users don't need to manually rename files.

For more history/details/examples, please see RedHat case 03309188

@Paul The problem is that there's a whole other (and seemingly large) set of users who upload a file with a given name and want to be able to "dnf install" using exactly that name, and consider it a bug that they cannot do so.  See https://bugzilla.redhat.com/show_bug.cgi?id=2162591

Automatic renaming is not compatible with that, because of course whatever you suggested the name should be originally might be ignored.

However I do hear your concern about the UX not being ideal - we should address that, and something along the lines of your backup suggestion is entirely reasonable.

I think you are mistaken on a few points.

"dnf install <name>" matches the RPM name listed in the yum repository metadata (which in turn is extracted from the metadata within the RPM).  It does NOT match the filename of the uploaded RPM file.
For example, in https://bugzilla.redhat.com/show_bug.cgi?id=2162591 the filename is "jdk-8u361-linux-x64.rpm", but the RPM metadata within that file includes "name: jdk1.8", "epoch: 2000", "version: 1.8.0_361", "release: fcs".
Regardless of whether or not the file is renamed by Satellite/Pulp, you cannot run "dnf install jdk" or "dnf install jdk-8u361" or "dnf install "jdk-8u361-linux-x64" or anything similar.  You can only run "dnf install jdk1.8" or "dnf install jdk1.8-2000:1.8.0_361-fcs" or similar.  Even if Satellite/Pulp preserves the filename, I don't believe there are any command line arguments to DNF that will operate on any part of that "jdk-8u361-linux-x64.rpm" filename; DNF will only operate using values that match the RPM metadata.

I believe the only things that DNF uses the RPM filename for are the URL that DNF uses to retrieve the file, and the filename that is written if you run `dnf download ...`
I would be surprised if any users care what the URL is as long as DNF works properly when using it.  (Note that the whole problem here is that preserving the original filename causes Pulp and/or DNF to misbehave in a number of cases.  Automatically renaming the file would avoid all of that.)
I would also be surprised if anyone would prefer a mal-formed filename to be preserved by `dnf download ...`, as opposed to being replaced with a well-formed filename.

My read of https://bugzilla.redhat.com/show_bug.cgi?id=2162591 is that they are NOT asking for the filename to be preserved.  Rather, they are asking the same thing that I am - For Satellite to restore the Pulp 2 behavior that renamed the file automatically, since Pulp 2 was working fine for their use case but now Pulp 3 is not.

I'll admit that I don't follow Pulp bug reports closely, so I could be missing something.  However, I have not seen anyone state that they consider automatic RPM file renaming to be a bug, and I am not aware of any code paths or use cases where preserving the original filename would be preferable over automatically renaming the RPM file.
Note that Satellite has been automatically renaming RPMs for decades (all versions of Satellite 5 renamed RPMs, and Satellite 6 did up until 6.10), so even if preservation of the original filename is considered a "new feature", it shouldn't be rolled out until all of the existing use cases can be addressed without major regressions (like the regressions we are seeing here and in https://bugzilla.redhat.com/show_bug.cgi?id=2161993 and https://bugzilla.redhat.com/show_bug.cgi?id=2162591).

Created attachment 1954920 [details]
Aggregated upstream fixes for Satellite 6.12.2

Upstream fixes:
https://github.com/pulp/pulp_rpm/issues/3039
https://github.com/Katello/katello/pull/10329

To apply to Satellite 6.12.2:
ssh satellite
sudo -s
patch -p0 < fix-2151657.patch
satellite-maintain service restart

That will fix new uploads, but will not fix existing uploads.

To attempt to fix some existing uploads that were broken due to an epoch in location_href, I tried:
sudo -u postgres psql pulpcore
select location_href from rpm_package where location_href LIKE '%:%';
update rpm_package set location_href='...' where location_href='...';
\q
satellite-maintain service restart
# Re-publish and promote Content Views.

However, that didn't seem to fix the published repository metadata, so DNF is still failing to download the relevant RPMs.

Any suggestions for how to fix the existing uploads, other than deleting and re-uploading them?

Hi

> That will fix new uploads, but will not fix existing uploads.
> 
> To attempt to fix some existing uploads that were broken due to an epoch in
> location_href, I tried:
> sudo -u postgres psql pulpcore
> select location_href from rpm_package where location_href LIKE '%:%';
> update rpm_package set location_href='...' where location_href='...';
> \q

I suggest you to fix the relative_path (which has the wrong format of filename) instead of fixing the location_href.

> satellite-maintain service restart
> # Re-publish and promote Content Views.
> 
> However, that didn't seem to fix the published repository metadata, so DNF
> is still failing to download the relevant RPMs.
> 
> Any suggestions for how to fix the existing uploads, other than deleting and
> re-uploading them?

You need to force regenerate metadata of the default organization view custom repository and then publish and promote new version of any related content views.


Do you have a support case open? If you need further support, I suggest you to raise a Red Hat support case and refer to this bugzilla.

For anyone else following along:

If you uploaded an RPM that did not have the expected filename at upload time, then you need to manually fix relative_path:
sudo -u postgres psql pulpcore
select relative_path from core_contentartifact where relative_path LIKE 'fubar%';
update rpm_package set relative_path='...' where pulp_id='...';
\q

If you uploaded an RPM that had an epoch set in its metadata (whether or not it was uploaded with the proper filename), then you need to manually fix location_href and filename:
sudo -u postgres psql pulpcore
select location_href from rpm_package where location_href LIKE '%:%';
update rpm_package set location_href='...' where location_href='...';
\q
sudo -u postgres psql foreman
select filename from katello_rpms where filename LIKE '%:%';
update katello_rpms set filename='...' where filename='...';
\q

After that, you need to run `satellite-maintain service restart` (or reboot the Satellite server) to ensure that all of the processes pick up the new DB data.

Reading the code, it looks to me like you should simply be able to publish/promote new Content Views to get these updated values put in all the right places.  However, simply publishing/promoting didn't actually seem to fix everything for me.

Things did start working for me after I republished the metadata of the standalone repositories where the RPMs were uploaded (https://access.redhat.com/solutions/6663481) and then published/promoted the Content Views again.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.