Today I deleted a mirror of rhel-6, and then recreated it with the exact same settings. I had not synced the deleted rhel-6 in a while (days). After creating the new rhel-6 and syncing, I was unable to publish. rbarlow@guava ~/d/p/pulp> pulp-admin rpm repo create --repo-id rhel-6-server --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os --feed-ca-cert=/home/rbarlow/redhat-uep.pem --feed-cert=/home/rbarlow/****.pem --feed-key /home/rbarlow/****-key.pem rbarlow@guava ~/d/p/pulp> pulp-admin rpm repo publish run --repo-id rhel-6-server +----------------------------------------------------------------------+ Publishing Repository [rhel-6-server] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Publishing Distribution files [-] ... completed Publishing RPMs [ ] 0% 0 of 12527 items Task Failed Pulp exception occurred: PulpExecutionException Here is the relevant bit of the logs: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[a251eaff-c9de-4f5e-ad85-3356a5dc4c3d] celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_release_resource[64c5cbfc-d6ff-4723-ba31-890767020610] celery.worker.job:INFO: Task pulp.server.tasks.repository.sync_with_auto_publish[24ca3aad-52b5-4eb9-a163-bc15facec00c] succeeded in 876.964413444s: <pulp.server.async.tasks.TaskResult object at 0x31df210> celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8b06bb8f-bc50-4c54-b899-353f1328c8c2] celery.worker.job:INFO: Task pulp.server.async.tasks._queue_release_resource[c8c20ce2-fdeb-46f3-9b5d-e991a524f610] succeeded in 0.0379554569954s: None celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8b06bb8f-bc50-4c54-b899-353f1328c8c2] succeeded in 0.0232481429994s: None celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[fea8da63-dbad-401c-aa00-519d3bc54778] succeeded in 1.16808687799s: None py.warnings:WARNING: /usr/lib/python2.7/site-packages/celery/app/control.py:34: DuplicateNodenameWarning: Received multiple replies from node name True. py.warnings:WARNING: Please make sure you give each node a unique nodename using the `-n` option. py.warnings:WARNING: W_DUPNODE.format(dup), py.warnings:WARNING: celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[c3d6abb1-e132-407d-a61f-a3247b945ab1] succeeded in 1.29009989899s: None celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[55ae9f20-028d-43fe-b12a-914ee50d31d3] succeeded in 1.233596609s: None celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[cdfe3d60-78c5-409e-867b-b09ee64268a6] succeeded in 1.280725091s: None pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [rhel-6-server] pulp.server.managers.repo.publish:ERROR: Traceback (most recent call last): pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp/server/pulp/server/managers/repo/publish.py", line 126, in _do_publish pulp.server.managers.repo.publish:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp/server/pulp/server/async/tasks.py", line 543, in wrap_f pulp.server.managers.repo.publish:ERROR: return f(*args, **kwargs) pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/distributor.py", line 130, in publish_repo pulp.server.managers.repo.publish:ERROR: return self._publisher.publish() pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 169, in publish pulp.server.managers.repo.publish:ERROR: step.process() pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 368, in process pulp.server.managers.repo.publish:ERROR: self._process_block() pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 603, in _process_block pulp.server.managers.repo.publish:ERROR: self.process_unit(package_unit) pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/publish.py", line 157, in process_unit pulp.server.managers.repo.publish:ERROR: self._create_symlink(source_path, destination_path) pulp.server.managers.repo.publish:ERROR: File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 427, in _create_symlink pulp.server.managers.repo.publish:ERROR: raise RuntimeError(msg % {'s': source_path}) pulp.server.managers.repo.publish:ERROR: RuntimeError: Will not create a symlink to a non-existent source [/var/lib/pulp/content/rpm/389-ds-base/1.2.10.2/15.el6/x86_64/e0c7dbd9b0d6ef7c1cee92bc3e6d8a41e5f4df3e/389-ds-base-1.2.10.2-15.el6.x86_64.rpm]
I looked at this in some detail over the past few days and with bcourt. I was unable to reproduce this with the following scenario: * install pulp 2.3 * sync a repo that contains a Packages dir[1] * upgrade to 2.4 * publish repo [1] note that the repo can't have a .treeinfo with repomd.xml defined right now due to #1099600 I tried some other upgrade/sync/publish combinations as well which did not cause issue. Additionally, Barnaby and I were unable to find how the scenario in comment #1 would happen during review of the code in question. Perhaps #1095829 fixed the stack you saw? Marking as closed/worksforme, but feel free to re-open if this reappears.
Possible steps to reproduce: Sync a repo and stop pulp services during syncing or repo generation. Then start again. The customer is doing daily full backups and syncing was still running we have found. This could be issue also with new pulp versions, could you please test this?
Pulp no longer uses this bug tracker (where Product=Pulp). I just happened to find this one in the NEW state while looking through Bugzilla. It should be in CLOSED state. I'm not sure what the right resolution is so I'm setting it back to what it was (WORKSFORME). If you want to refile it in upstream Pulp please file here: https://pulp.plan.io/projects/pulp/issues/new