Bug 1679736
| Summary: | Incremental export of repository does not create proper repodata. | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Suraj Patil <supatil> | ||||||
| Component: | Inter Satellite Sync | Assignee: | satellite6-bugs <satellite6-bugs> | ||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Lai <ltran> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 6.3.5 | CC: | bbuckingham, cduryee, daviddavis, ehelms, jturel, lee.huynh2, ltran, mkalyat, paji, sadas | ||||||
| Target Milestone: | 6.10.0 | Keywords: | Triaged | ||||||
| Target Release: | Unused | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2021-07-16 14:59:33 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Suraj Patil
2019-02-21 18:14:16 UTC
Created attachment 1537150 [details]
Initial repository import
Created attachment 1537151 [details]
After incremental import
I'm not sure this is a Pulp bug. Katello, can you check and perhaps give us a reproducer against Pulp? I found an unofficial way to get around to resync the whole content using createrepo command. Here is the procedure: 1. Go to repodata directory and take a note of checksum type in repomd.xml file # cd ../6Server/x86_64/os/repodata # grep "checksum type" repomd.xml 2. Run createrepo # createrepo -s sha1 ../6Server/x86_64/os 3. Change back permission # chmod 755 -R repodata 4. Doing the normal synchronization (In reply to Suraj Patil from comment #0) > Description of problem: > > When we do the incremental export of the repository, it creates the repodata > only for the content(rpms) which is included in the incremental export. This > repodata is useless because when we copy the exported content to the > disconnected satellite where previous/initial content is present it > overrides the repomd file which then points to the new repodata which does > not have the details of previous content. > When you copy the data over, do you copy the new directory on top of the old one, or do you move the old one first? We have to copy the new directory on top of the old one. I don't know other way because we have incrementally synced every week. Doing your way will have to move the old one every time the incremental export is not a file-level incremental that can be copied on top of existing repodata. It is a set of repos that contains just the RPMs that have been synced since the last export. For example, if you had rpms A and B and performed an export, then synced and got a new rpm C and performed another export with a date before C but after A and B, you'd get a yum repo with just C in it. You should be able to sync this into the disconnected Satellite with "mirror on sync" disabled and the disconnected satellite would then have A, B and C. If you copied the new export repo with just C on top of the old export, you'd get a repo with A, B and C (since A and B were not deleted) but the repodata would only be aware of C. Are you looking for more of a file-level delta that can be applied to an existing repo tree? The real problem is when we rebuild the disconnected satellite server. All repos data are in /var/lib/import which is under /var/lib filesystem and are intact. However, after satellite software is reinstalled, we try to sync the existing repos and it syncs only the most recent incrementally exported data which is rpm C. Therefore, to make it to sync all A,B,C, we have to run createrepo on every repo. This will create a new repomd.xml file that includes all A,B,C. Hello Chris, What you said is correct, If I copy the new export over the old all the packages are intact but the issue is with repodata. At this point repodata only contains C package. I am aware of "mirror on sync" option, and till the point, I remember in my satellite "mirror on sync" was disabled and still after the import satellite was only having the repodata of package C. I will try to reproduce this again just to confirm "mirror on sync" option and will let you know. Hello, I have reproduced it again and confirmed that if "mirror on sync" is disabled and this time satellite say new 731 packages found and In repository details it shows 21076(old package count) + 731(New packages) = 21807. This means that "mirror on sync" option works. But not sure what I did wrong because newly exported incremental repository, show package count 23927 in repodata. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [root@satellite 1554824079.59]# zcat /var/www/html/pub/sat-import/Example/Library/content/dist/rhel/server/7/7Server/x86_64/os/repodata/c5510a36e3ee920a1901a716d070c71f06303cd5-primary.xml.gz | head <?xml version="1.0" encoding="UTF-8"?> <metadata packages="23927" xmlns="http://linux.duke.edu/metadata/common" xmlns:rpm="http://linux.duke.edu/metadata/rpm"><package type="rpm"> <name>openssl-devel</name> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ So total package count should be 23927. I still ask the customer if he is using "mirror on sync" option. Satellite 6.10 introduces Pulp 3. With that change, the export/import capabilities of Satellite have been re-written and enhanced to enabled support for both full as well as incremental export/import. The support for incremental was verified with bug 1944733; therefore, closing this bugzilla as a duplicate. If there are any questions or concerns, please do let us know. Thanks! *** This bug has been marked as a duplicate of bug 1944733 *** The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |