Description of problem: Assume due to a network issue, some package / docker image / puppet module / whatever pulp unit gets corrupted during downloading to Satellite6. Currently, there is no simple way how to re-fetch that pulp unit. I.e. deleting it from /var/lib/pulp/content and re-syncing the repo, nothing will happen since just repo metadata are compared and only the difference is fetched. The only workaround in this situation is to 1) disable the affected repo, 2) foreman-rake katello:delete_orphaned_content, 3) enable repo, 4) sync it (and hope no other unit to be malformed). That is not too much practical. Please allow pulp to forcefully re-sync whole repo. Version-Release number of selected component (if applicable): Sat 6.1.8 How reproducible: 100% Steps to Reproduce: 1. Sync a repo 2. Manually damage whatever downloaded file (RPM, docker image,..) 3. Try to fix it by pulp operations/ Actual results: 3. No such operation exists. Expected results: 3. pulp to have such op. Additional info:
Also capsules should have this feature..
Per 6.3 planning, moving out non acked bugs to the backlog
*** Bug 1314569 has been marked as a duplicate of this bug. ***
Pulp supports this through the "on-demand" feature set. If a repository has been sync'd with a download policy of "on_demand" or "background", then you can run the "download_repo" task with verification on. From the docs, that will "check all units in the repository for corrupted or missing files and re-download files as necessary". http://docs.pulpproject.org/dev-guide/integration/rest-api/repo/sync.html#download-a-repository
Michael, So it does NOT support it if the repository is set to immediate?
Also, for on_demand and background, wouldn't it cause the full repo contents to be downloaded? In the on_demand case that wouldn't be desired.
(In reply to Michael Hrivnak from comment #10) > Pulp supports this through the "on-demand" feature set. If a repository has > been sync'd with a download policy of "on_demand" or "background", then you > can run the "download_repo" task with verification on. From the docs, that > will "check all units in the repository for corrupted or missing files and > re-download files as necessary". > > http://docs.pulpproject.org/dev-guide/integration/rest-api/repo/sync. > html#download-a-repository Seems we have some restrictions and it only works for "on-demand" which might not be accepted by the customer. Another point, how would you trigger the "download_repo" task with verification on in Satellite 6?
Correct, this does not work with a policy of "immediate". And for on_demand, that's correct. If you run the "download_repo" task, it will do exactly what it sounds like it would do. :) The question is: what would be desired in this case? Back to the original description of handling a corrupt file, the user can just delete it and be done. The next time it's requested by a client, pulp will re-download it.
Will pulp re-download it even in the case of an 'immediate' repo? The purpose is when multiple files are corrupted. For the user to even know there is corruption they have to wait for some client to try to download it. The desired behavior would be that for repositories of any download policy, scan all downloaded rpms, verify the checksum, and if its corrupt or missing, redownload the file.
pulp will not re-download in the case of an "immediate" repo. If that is needed, it would be a new feature. It would be difficult to do in a generic way, because for an "immediate" repo, the only opportunity to re-download a file is during a sync operation. In pulp 3, because of the database guarantees we'll get from postgres, we'll be able to make a full guarantee that if a unit is in the DB and its file is marked downloaded, its file will have been verified in-place. That doesn't help with the presumably-much-more-rare bit rot problem, but perhaps it's useful to prioritize that separately. If we're worried about files going corrupt while at rest, we should consider using an external tool or filesystem that's purpose-built to watch files for corruption. Then we would still need an easy path to recovery for that file. Perhaps the easiest approach in the mean time would be to make it easy to remove a unit completely from pulp, re-sync a repo whose remote feed contains it, and then copy it back into all of the repos where it was present. We could look at how easy it is or isn't to figure out all of the repos a unit is in, and then katello could orchestrate the removal and re-sync. Does that sound worth pursuing? It's also worth considering replacing use of the "immediate" download policy with "background" always. That would make a lot of this easier, and simplify many other things as well.
Use case I would think of: - The repository is corrupt - we don't trust it All we want is a full (brute force) sync to rebuild it. - Sync just one RPM If we have a corrupt file it is not always straight forward to know which file we should remove to have it re-synced - In my case the the repmod.xml files were not created Would the solutions mentioned recreate them?
(In reply to Michael Hrivnak from comment #16) > pulp will not re-download in the case of an "immediate" repo. If that is > needed, it would be a new feature. It would be difficult to do in a generic > way, because for an "immediate" repo, the only opportunity to re-download a > file is during a sync operation. > > In pulp 3, because of the database guarantees we'll get from postgres, we'll > be able to make a full guarantee that if a unit is in the DB and its file is > marked downloaded, its file will have been verified in-place. > > That doesn't help with the presumably-much-more-rare bit rot problem, but > perhaps it's useful to prioritize that separately. If we're worried about > files going corrupt while at rest, we should consider using an external tool > or filesystem that's purpose-built to watch files for corruption. Then we > would still need an easy path to recovery for that file. > > Perhaps the easiest approach in the mean time would be to make it easy to > remove a unit completely from pulp, re-sync a repo whose remote feed > contains it, and then copy it back into all of the repos where it was > present. We could look at how easy it is or isn't to figure out all of the > repos a unit is in, and then katello could orchestrate the removal and > re-sync. Does that sound worth pursuing? > > It's also worth considering replacing use of the "immediate" download policy > with "background" always. That would make a lot of this easier, and simplify > many other things as well. I have the same use case as Ilan in https://bugzilla.redhat.com/show_bug.cgi?id=1330042#c17. But I also think we should consider https://bugzilla.redhat.com/show_bug.cgi?id=1223023 which tries to address the reason for packages/repositories not being downloaded correctly (with 0 byte size or truncated). If we can better manage that the content that is downloaded is correct and not corrupt somehow the need for a re-sync becomes less important. Still I think that the feature might be helpful in certain situation as the functionality was also something customers were using in Satellite 5.
Michael, I think a workaround might be to flip the repo from immediate to background and sync. That would work? The user could then flip back to immediate. I don't think its appropriate at this time to change immediate to background for all repos (if we had implemented this when we implemented lazy that would be one thing, but i really don't want to make this change now). We should get together and discuss the behavior in pulp 3.0 to ensure the behavior is consistent from what is expected. The lack of this feature is extremely embarrassing and i don't want it to go without it any longer than we need to.
(In reply to Michael Hrivnak from comment #16) > Perhaps the easiest approach in the mean time would be to make it easy to > remove a unit completely from pulp, re-sync a repo whose remote feed > contains it, and then copy it back into all of the repos where it was > present. We could look at how easy it is or isn't to figure out all of the > repos a unit is in, and then katello could orchestrate the removal and > re-sync. Does that sound worth pursuing? I was thinking about possible workaround in this direction (for traditional "immediate" type of repo - to have the workaround applicable to 6.1 customers as well): (I restrict the workaround to fix RPMs only): 1) Identify all units in the repo: pulp-admin rpm repo content rpm --repo-id <repo_id> --fields filename 2) Compare to /var/lib/pulp/published/yum/master/yum_distributor/<repo_id>/*/*rpm 3) If some RPM from the repo is missing in the dir, remove it as unit from the repo: pulp-admin rpm repo remove rpm --repo-id <repo_id> --str-eq="filename=<filename>" 4) sync the repo: pulp-admin rpm repo sync run --repo-id <repo_id> 5) Will the above work? Will it download the missing units (despite they were added to CDN a long time ago)?
I'm not clear if the file is corrupted in the backend storage of Pulp or just in this 1 published repo. For example if this unit is in 3 repos is the unit in all repos corrupted or just 1? If it's just the 1 repo then your workaround looks just right. If it's all repos that refer to the unit then you need to remove the unit from all repos not just the 1. This effectively orphans the unit. After that you need to do an orphan cleanup which will remove the DB and filesystem representation of it. Then any repo you sync which contains the unit will receive it again on next sync. Note that if the unit is copied to a repo, you would have to re-copy that unit again. Does that make sense?
As Brian notes, you would need to do an orphan remove after step 3. And if that unit is associated with any other repositories, you would have to remove it from them also before doing the remove. Otherwise, the same unit would just get re-associated to the repo when you do the sync.
(In reply to Michael Hrivnak from comment #24) > As Brian notes, you would need to do an orphan remove after step 3. And if > that unit is associated with any other repositories, you would have to > remove it from them also before doing the remove. > > Otherwise, the same unit would just get re-associated to the repo when you > do the sync. FYI maybe I am solving a different problem but unassociating the unit from all repos and clearing orphans would delete the RPM from /var/lib/pulp/content. And subsequent pulp repo sync will *not* fetch the RPM again and will do nothing. But if I: - identify the packages missing in the repo (the base one, like MyOrganization-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server) - one by one, remove them from the repo - one by one, re-add them back - sync the repo (and forcefully re-publish it (via http distributor)) - re-publish content views containing this repo (in pulp terminology: re-publish repos having this repo as one of importers) See https://access.redhat.com/solutions/2653831 for details (and blame mongo why the script is slow and harder to follow).
OK.. coming back to this. the use case seems simple to the end user: Sync --force where the user is same: "To heck with what you think you have, redownload it". Is this on hte pulp roadmap some place?
*** This bug has been marked as a duplicate of bug 1223023 ***
Closing this as a dupe, and I will change the title of the other to be more clear.