Bug 1575193 - File repo type does not handle updates to files on manifest during re-sync correctly
Summary: File repo type does not handle updates to files on manifest during re-sync co...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.3.1
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: 6.4.0
Assignee: satellite6-bugs
QA Contact: Peter Ondrejka
URL:
Whiteboard:
Depends On: 1452212
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-05 03:38 UTC by Craig Donnelly
Modified: 2021-04-06 17:50 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1452212
Environment:
Last Closed: 2018-10-16 19:11:13 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 3100 0 Normal CLOSED - CURRENTRELEASE Removal of existing iso units doesn't work if there are multiple iso files 2019-05-16 15:32:39 UTC

Description Craig Donnelly 2018-05-05 03:38:48 UTC
+++ This bug was initially created as a clone of Bug #1452212 +++

Description of problem:

Currently, if you generate a PULP_MANIFEST and sync from another location via URL, you will receive the files in the Manifest and they will become locally available on the Satellite, but if you UPDATE a file that has already been imported and regenerate the PULP_MANIFEST, then sync again, the Satellite's versions of the manifest will have a duplicate file name entry with separate sha256sums and sizes, but will not retrieve the new file (It will maintain the original in the repo).

Version-Release number of selected component (if applicable):
nightly

How reproducible:
100%

Steps to Reproduce:
1. Generate PULP_MANIFEST and sync a File repo from a http based URL
2. Make a change to a file (text file, etc..), but DO NOT change the file NAME.
3. Re-Generate the PULP_MANIFEST for the remote file repo, and then re-sync on Satellite.

Actual results:
File that was modified will have two entries on the PULP_MANIFEST inside the local Satellite copy of the repo with separate sha256sums and sizes, but identical name, and only the original copy of the file prior to editing will exist in the repo.

Expected results:
We should replace original file with the newest checksum. (Act like mirror-on-sync?)

Additional info:
Perhaps the File repo type should be able to determine the time of PULP_MANIFEST generation and make a determination about what should exist based off of that? (i.e. Include a date inside the PULP_MANIFEST on creation)

--- Additional comment from Michael Hrivnak on 2017-05-19 15:24:53 EDT ---

Can you add the specific Pulp version that was in use? Thanks!

--- Additional comment from pulp-infra@redhat.com on 2017-05-19 15:37:02 EDT ---

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-05-19 15:37:05 EDT ---

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

--- Additional comment from Craig Donnelly on 2017-05-19 15:41:03 EDT ---

Michael,

During this test I had:

pulp-server-2.9.3-2.el7sat.noarch

--- Additional comment from pulp-infra@redhat.com on 2017-05-23 11:04:05 EDT ---

The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-06-19 15:31:57 EDT ---

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-06-22 15:32:06 EDT ---

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-07-03 11:04:53 EDT ---

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-07-03 11:33:45 EDT ---

All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

--- Additional comment from pulp-infra@redhat.com on 2017-07-20 13:32:20 EDT ---

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-07-20 14:02:05 EDT ---

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

--- Additional comment from Lukáš Hellebrandt on 2017-09-26 08:49:59 EDT ---

FailedQA with Sat6.3, snap 17. Thanks for advice to rplevka.

As manipulating Pulp directly is not supported is Satellite, we can use the following reproducer:

1) Create Product
2) In Product, create Repo1 with empty URL
3) Upload File to Repo1
4) In Product, create Repo2 with URL=<content_url_of_repo_1> [1]
5) Sync Repo2
6) Change contents of File
7) In any of Repo1 or Repo2 (works both ways), upload File
8) Sync Repo2 again
9) In content URL of Repo2, view contents of PULP_MANIFEST file

Actual results:
Two entries with the same filename and different hash

Expected results:
Not sure... how do we decide which version of the file to keep?


[1] for example, http://<FQDN>/pulp/isos/e2f04116-1ce6-4390-a781-75d00299c57f/

--- Additional comment from pulp-infra@redhat.com on 2017-09-26 09:01:59 EDT ---

Requesting needsinfo from upstream developer daviddavis@redhat.com because the 'FailedQA' flag is set.

--- Additional comment from David Davis on 2017-09-28 15:31:10 EDT ---

Found the bug and opened a new upstream issue.

--- Additional comment from pulp-infra@redhat.com on 2017-09-28 15:32:20 EDT ---

Requesting needsinfo from upstream developer daviddavis@redhat.com because the 'FailedQA' flag is set.

--- Additional comment from pulp-infra@redhat.com on 2017-09-28 16:37:39 EDT ---

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-09-28 16:37:43 EDT ---

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-10-03 13:01:29 EDT ---

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

--- Additional comment from Tanya Tereshchenko on 2017-10-03 17:21:08 EDT ---

Upstream bug is fixed and merged, moving to POST

--- Additional comment from Lukáš Hellebrandt on 2017-10-17 04:45:21 EDT ---

Verified with Sat6.3 snap 20.

Used reproducer from comment 12. After that, there were only the ew file's metadata and only the new file in both repositories.

--- Additional comment from pulp-infra@redhat.com on 2017-10-19 10:01:56 EDT ---

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-10-25 17:32:07 EDT ---

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-10-26 11:01:58 EDT ---

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-10-26 19:31:53 EDT ---

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-10-30 16:31:38 EDT ---

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

--- Additional comment from pulp-infra@redhat.com on 2017-11-16 15:32:38 EST ---

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

--- Additional comment from pm-sat@redhat.com on 2018-02-21 11:54:37 EST ---

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336




---------------------------------------------------------------------------

In the process of working through BZ 1505387, I was able to reproduce this issue on Satellite 6.3.1-3.

Comment 1 Craig Donnelly 2018-05-05 03:52:04 UTC
Actions taken to reproduce:

# cd /var/www/html/pub
# mkdir file_repo
# cd file_repo

# rpm -q python-pulp-manifest
python-pulp-manifest-2.13.4.8-1.el7sat.noarch

# for i in 1 2 3;do echo "File$i" >> file$i.txt;done
# ll
total 12
-rw-r--r--. 1 root root 6 May  5 05:23 file1.txt
-rw-r--r--. 1 root root 6 May  5 05:23 file2.txt
-rw-r--r--. 1 root root 6 May  5 05:23 file3.txt
# md5sum file*
f3b203c69dabc6fc8029764a3d5ee1e0  file1.txt
cd71eba7bb4a821022ec1c2004a2b1ae  file2.txt
f1dae39ae2855d00f2299f341094de77  file3.txt
# cat file*
File1
File2
File3
# pulp-manifest .
# ll
total 16
-rw-r--r--. 1 root root   6 May  5 05:23 file1.txt
-rw-r--r--. 1 root root   6 May  5 05:23 file2.txt
-rw-r--r--. 1 root root   6 May  5 05:23 file3.txt
-rw-r--r--. 1 root root 231 May  5 05:23 PULP_MANIFEST
# cat PULP_MANIFEST 
file1.txt,ebea1137c5ece3f8a58f0e1a0da1411fe0a2648501419d190b3b154f3f191259,6
file2.txt,d111177254889051153a91fa445b85a19144f6fd63624393df6d1e9066406f62,6
file3.txt,3be4cac02ac5009c10286b6bdbd1b8e9eb9cb11a33344112d594acc1b1645e76,6

At this point, create custom product + custom repository:
http://sat.fqdn/pub/file_repo
Sync said repo.

First results on Satellite are great, everything as expected: i.e. PULP_MANIFEST in Satellite repo looks fine and like the one that was manually generated.

Now changes the files (not add, but change):

# for i in 1 2 3;do echo "File$i Updated" > file$i.txt;done
# ll
total 16
-rw-r--r--. 1 root root  14 May  5 05:30 file1.txt
-rw-r--r--. 1 root root  14 May  5 05:30 file2.txt
-rw-r--r--. 1 root root  14 May  5 05:30 file3.txt
-rw-r--r--. 1 root root 231 May  5 05:23 PULP_MANIFEST
# cat file*
File1 Updated
File2 Updated
File3 Updated
# md5sum file*
3dbcecec63d35c032f2a1a17d6fe25a2  file1.txt
ec9416093a9503863ec2a68ff00b1bcb  file2.txt
19ac24bd99dea28dad195967af9cc001  file3.txt
# cat PULP_MANIFEST 
file1.txt,ebea1137c5ece3f8a58f0e1a0da1411fe0a2648501419d190b3b154f3f191259,6
file2.txt,d111177254889051153a91fa445b85a19144f6fd63624393df6d1e9066406f62,6
file3.txt,3be4cac02ac5009c10286b6bdbd1b8e9eb9cb11a33344112d594acc1b1645e76,6
# pulp-manifest .
# cat PULP_MANIFEST 
file1.txt,582bd05dcf5d75d823c503066e046eb8bfa3df87647d157d709801fe410e8c52,14
file2.txt,84a0ecc2cb21bd9d47cd059a7ba2dbc5cf4259b06fec2016d89226ec0774bf73,14
file3.txt,c84e33433db61cef58a5932946ddf292320b8a5c830b7e5928e0336bc2f89196,14

This manually generated PULP_MANIFEST is fine. Now sync repo again, and review PULP_MANIFEST inside satellite:

http://sat.fqdn/pulp/isos/Default_Organization/Library/custom/Custom_Repos/file_repo/PULP_MANIFEST:

file1.txt,ebea1137c5ece3f8a58f0e1a0da1411fe0a2648501419d190b3b154f3f191259,6
file2.txt,d111177254889051153a91fa445b85a19144f6fd63624393df6d1e9066406f62,6
file2.txt,84a0ecc2cb21bd9d47cd059a7ba2dbc5cf4259b06fec2016d89226ec0774bf73,14
file1.txt,582bd05dcf5d75d823c503066e046eb8bfa3df87647d157d709801fe410e8c52,14
file3.txt,c84e33433db61cef58a5932946ddf292320b8a5c830b7e5928e0336bc2f89196,14

Depending on the order of how that broken file gets generated, sometimes the files in the repo actually DO NOT GET UPDATED.

Comment 3 pulp-infra@redhat.com 2018-05-05 22:32:52 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2018-05-05 22:32:56 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2018-05-05 22:33:30 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2018-05-05 22:33:35 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2018-05-05 23:02:47 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 9 pulp-infra@redhat.com 2018-05-07 12:02:51 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2018-05-07 12:02:56 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2018-05-07 12:32:43 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 16 Peter Ondrejka 2018-07-10 09:19:57 UTC
Verified on Sat 6.4 snap 11, pulp manifest is correctly updated after resync of edited file content

Comment 17 Bryan Kearney 2018-10-16 19:11:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927

Comment 18 pulp-infra@redhat.com 2019-05-16 15:32:40 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.


Note You need to log in before you can comment on or make changes to this bug.