Description of problem:
I wish I knew exact repro steps. I am not sure how often this can/will take place, but I was doing nothing exotic.
Doing...something... causes pulp to get confused and be unable to manage a given (custom?) repo. This in turn blocks dynflow from completing anything else content-related subsequently. I will include what i THINK are the repro steps.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Create a custom product and a custom repo. Put a bad URL as the repo or something, try to sync it (imagine you fat-fingered a repo or something)
2. Delete the bad repo and create a new one, probably with the same name.
3. Attempt to sync??
Jul 25 10:01:04 cloud-qe-4 pulp: pulp.server.webservices.middleware.exception:ERROR: Duplicate resource: Default_Organization-snap2-tools
No real error is thrown in UI or on screen, but if user tries to do anything else content-related, it will all get backed up , waiting for something to complete from earlier. I tried to sync RHEL content about 4 times, wondering why it would simply say "Cancelled" after 5 seconds, only to find a half-dozen or more blocked tasks in dynflow.
I wish I knew the exact issue so I could provide an expected result. But it's evident pulp is trying to CRUD some sort of resource that already exists, even though it shouldn't.
For the tasks being stucked, I was not able to reproduce the issue with this steps, however, I hit a situation, where the repository create was stuck because pulp was not picking up the tasks for it, because some other tasks were happening in Pulp (such as repository syncing).
More details here https://bugzilla.redhat.com/show_bug.cgi?id=1124570
For the duplicate error, it doesn't seem to be the cause of the tasks being stucked, but the issue on Katello side is allowing the create the new repo with the same name while the deletion of the old one was not finished yet (because, in this case, the repository was stuck on the deletion in Pulp)
For the dupicated error, I've filed separate issue here https://bugzilla.redhat.com/show_bug.cgi?id=1129635
I wasn't able to reproduce with the given steps but I was able to reproduce what I think is the problem by syncing large repos (one repo for every pulp worker). Talking with Ivan, we may need to talk about how to handle heavy usage where the pulp workers are overloaded.
The duplicate errors look like they're caused by katello trying to create a repository in pulp that already exists. I believe pulp is functioning correctly.
Given there are no reproducer steps (and can't be easily reproduced), we close this bug. Also, there was a lot of changes on more layers involved in this bug that could just fix the problem. If anyone feels the issue is still there as described in this BZ, please reopen or even better file a new bug with relevant information from recent versions.