+++ This bug was initially created as a clone of Bug #1452212 +++ Description of problem: Currently, if you generate a PULP_MANIFEST and sync from another location via URL, you will receive the files in the Manifest and they will become locally available on the Satellite, but if you UPDATE a file that has already been imported and regenerate the PULP_MANIFEST, then sync again, the Satellite's versions of the manifest will have a duplicate file name entry with separate sha256sums and sizes, but will not retrieve the new file (It will maintain the original in the repo). Version-Release number of selected component (if applicable): nightly How reproducible: 100% Steps to Reproduce: 1. Generate PULP_MANIFEST and sync a File repo from a http based URL 2. Make a change to a file (text file, etc..), but DO NOT change the file NAME. 3. Re-Generate the PULP_MANIFEST for the remote file repo, and then re-sync on Satellite. Actual results: File that was modified will have two entries on the PULP_MANIFEST inside the local Satellite copy of the repo with separate sha256sums and sizes, but identical name, and only the original copy of the file prior to editing will exist in the repo. Expected results: We should replace original file with the newest checksum. (Act like mirror-on-sync?) Additional info: Perhaps the File repo type should be able to determine the time of PULP_MANIFEST generation and make a determination about what should exist based off of that? (i.e. Include a date inside the PULP_MANIFEST on creation) --- Additional comment from Michael Hrivnak on 2017-05-19 15:24:53 EDT --- Can you add the specific Pulp version that was in use? Thanks! --- Additional comment from pulp-infra on 2017-05-19 15:37:02 EDT --- The Pulp upstream bug status is at NEW. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-05-19 15:37:05 EDT --- The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug. --- Additional comment from Craig Donnelly on 2017-05-19 15:41:03 EDT --- Michael, During this test I had: pulp-server-2.9.3-2.el7sat.noarch --- Additional comment from pulp-infra on 2017-05-23 11:04:05 EDT --- The Pulp upstream bug priority is at High. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-06-19 15:31:57 EDT --- The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-06-22 15:32:06 EDT --- The Pulp upstream bug status is at POST. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-07-03 11:04:53 EDT --- The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-07-03 11:33:45 EDT --- All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST. --- Additional comment from pulp-infra on 2017-07-20 13:32:20 EDT --- The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-07-20 14:02:05 EDT --- The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug. --- Additional comment from Lukáš Hellebrandt on 2017-09-26 08:49:59 EDT --- FailedQA with Sat6.3, snap 17. Thanks for advice to rplevka. As manipulating Pulp directly is not supported is Satellite, we can use the following reproducer: 1) Create Product 2) In Product, create Repo1 with empty URL 3) Upload File to Repo1 4) In Product, create Repo2 with URL=<content_url_of_repo_1> [1] 5) Sync Repo2 6) Change contents of File 7) In any of Repo1 or Repo2 (works both ways), upload File 8) Sync Repo2 again 9) In content URL of Repo2, view contents of PULP_MANIFEST file Actual results: Two entries with the same filename and different hash Expected results: Not sure... how do we decide which version of the file to keep? [1] for example, http://<FQDN>/pulp/isos/e2f04116-1ce6-4390-a781-75d00299c57f/ --- Additional comment from pulp-infra on 2017-09-26 09:01:59 EDT --- Requesting needsinfo from upstream developer daviddavis because the 'FailedQA' flag is set. --- Additional comment from David Davis on 2017-09-28 15:31:10 EDT --- Found the bug and opened a new upstream issue. --- Additional comment from pulp-infra on 2017-09-28 15:32:20 EDT --- Requesting needsinfo from upstream developer daviddavis because the 'FailedQA' flag is set. --- Additional comment from pulp-infra on 2017-09-28 16:37:39 EDT --- The Pulp upstream bug status is at POST. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-09-28 16:37:43 EDT --- The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-10-03 13:01:29 EDT --- The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug. --- Additional comment from Tanya Tereshchenko on 2017-10-03 17:21:08 EDT --- Upstream bug is fixed and merged, moving to POST --- Additional comment from Lukáš Hellebrandt on 2017-10-17 04:45:21 EDT --- Verified with Sat6.3 snap 20. Used reproducer from comment 12. After that, there were only the ew file's metadata and only the new file in both repositories. --- Additional comment from pulp-infra on 2017-10-19 10:01:56 EDT --- The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-10-25 17:32:07 EDT --- The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-10-26 11:01:58 EDT --- The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-10-26 19:31:53 EDT --- The Pulp upstream bug status is at POST. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-10-30 16:31:38 EDT --- The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug. --- Additional comment from pulp-infra on 2017-11-16 15:32:38 EST --- The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug. --- Additional comment from pm-sat on 2018-02-21 11:54:37 EST --- Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. > > > > For information on the advisory, and where to find the updated files, follow the link below. > > > > If the solution does not work for you, open a new bug report. > > > > https://access.redhat.com/errata/RHSA-2018:0336 --------------------------------------------------------------------------- In the process of working through BZ 1505387, I was able to reproduce this issue on Satellite 6.3.1-3.
Actions taken to reproduce: # cd /var/www/html/pub # mkdir file_repo # cd file_repo # rpm -q python-pulp-manifest python-pulp-manifest-2.13.4.8-1.el7sat.noarch # for i in 1 2 3;do echo "File$i" >> file$i.txt;done # ll total 12 -rw-r--r--. 1 root root 6 May 5 05:23 file1.txt -rw-r--r--. 1 root root 6 May 5 05:23 file2.txt -rw-r--r--. 1 root root 6 May 5 05:23 file3.txt # md5sum file* f3b203c69dabc6fc8029764a3d5ee1e0 file1.txt cd71eba7bb4a821022ec1c2004a2b1ae file2.txt f1dae39ae2855d00f2299f341094de77 file3.txt # cat file* File1 File2 File3 # pulp-manifest . # ll total 16 -rw-r--r--. 1 root root 6 May 5 05:23 file1.txt -rw-r--r--. 1 root root 6 May 5 05:23 file2.txt -rw-r--r--. 1 root root 6 May 5 05:23 file3.txt -rw-r--r--. 1 root root 231 May 5 05:23 PULP_MANIFEST # cat PULP_MANIFEST file1.txt,ebea1137c5ece3f8a58f0e1a0da1411fe0a2648501419d190b3b154f3f191259,6 file2.txt,d111177254889051153a91fa445b85a19144f6fd63624393df6d1e9066406f62,6 file3.txt,3be4cac02ac5009c10286b6bdbd1b8e9eb9cb11a33344112d594acc1b1645e76,6 At this point, create custom product + custom repository: http://sat.fqdn/pub/file_repo Sync said repo. First results on Satellite are great, everything as expected: i.e. PULP_MANIFEST in Satellite repo looks fine and like the one that was manually generated. Now changes the files (not add, but change): # for i in 1 2 3;do echo "File$i Updated" > file$i.txt;done # ll total 16 -rw-r--r--. 1 root root 14 May 5 05:30 file1.txt -rw-r--r--. 1 root root 14 May 5 05:30 file2.txt -rw-r--r--. 1 root root 14 May 5 05:30 file3.txt -rw-r--r--. 1 root root 231 May 5 05:23 PULP_MANIFEST # cat file* File1 Updated File2 Updated File3 Updated # md5sum file* 3dbcecec63d35c032f2a1a17d6fe25a2 file1.txt ec9416093a9503863ec2a68ff00b1bcb file2.txt 19ac24bd99dea28dad195967af9cc001 file3.txt # cat PULP_MANIFEST file1.txt,ebea1137c5ece3f8a58f0e1a0da1411fe0a2648501419d190b3b154f3f191259,6 file2.txt,d111177254889051153a91fa445b85a19144f6fd63624393df6d1e9066406f62,6 file3.txt,3be4cac02ac5009c10286b6bdbd1b8e9eb9cb11a33344112d594acc1b1645e76,6 # pulp-manifest . # cat PULP_MANIFEST file1.txt,582bd05dcf5d75d823c503066e046eb8bfa3df87647d157d709801fe410e8c52,14 file2.txt,84a0ecc2cb21bd9d47cd059a7ba2dbc5cf4259b06fec2016d89226ec0774bf73,14 file3.txt,c84e33433db61cef58a5932946ddf292320b8a5c830b7e5928e0336bc2f89196,14 This manually generated PULP_MANIFEST is fine. Now sync repo again, and review PULP_MANIFEST inside satellite: http://sat.fqdn/pulp/isos/Default_Organization/Library/custom/Custom_Repos/file_repo/PULP_MANIFEST: file1.txt,ebea1137c5ece3f8a58f0e1a0da1411fe0a2648501419d190b3b154f3f191259,6 file2.txt,d111177254889051153a91fa445b85a19144f6fd63624393df6d1e9066406f62,6 file2.txt,84a0ecc2cb21bd9d47cd059a7ba2dbc5cf4259b06fec2016d89226ec0774bf73,14 file1.txt,582bd05dcf5d75d823c503066e046eb8bfa3df87647d157d709801fe410e8c52,14 file3.txt,c84e33433db61cef58a5932946ddf292320b8a5c830b7e5928e0336bc2f89196,14 Depending on the order of how that broken file gets generated, sometimes the files in the repo actually DO NOT GET UPDATED.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.
Verified on Sat 6.4 snap 11, pulp manifest is correctly updated after resync of edited file content
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2927