Bug 1098340

Summary:	RuntimeError: Will not create a symlink to a non-existent source
Product:	[Retired] Pulp	Reporter:	Randy Barlow <rbarlow>
Component:	rpm-support	Assignee:	Chris Duryee <cduryee>
Status:	CLOSED WORKSFORME	QA Contact:	pulp-qe-list
Severity:	high	Docs Contact:
Priority:	medium
Version:	Master	CC:	bmbouter, lzap, skarmark
Target Milestone:	---	Keywords:	Reopened, Triaged
Target Release:	2.4.0
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-07-26 21:01:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Randy Barlow 2014-05-15 21:20:22 UTC

Today I deleted a mirror of rhel-6, and then recreated it with the exact same settings. I had not synced the deleted rhel-6 in a while (days). After creating the new rhel-6 and syncing, I was unable to publish.

rbarlow@guava ~/d/p/pulp> 
pulp-admin rpm repo create --repo-id rhel-6-server --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os --feed-ca-cert=/home/rbarlow/redhat-uep.pem --feed-cert=/home/rbarlow/****.pem --feed-key /home/rbarlow/****-key.pem

rbarlow@guava ~/d/p/pulp> pulp-admin rpm repo publish run --repo-id rhel-6-server
+----------------------------------------------------------------------+
                 Publishing Repository [rhel-6-server]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Publishing Distribution files
[-]
... completed

Publishing RPMs
[                                                  ] 0%
0 of 12527 items

Task Failed

Pulp exception occurred: PulpExecutionException

Here is the relevant bit of the logs:

celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[a251eaff-c9de-4f5e-ad85-3356a5dc4c3d]
celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_release_resource[64c5cbfc-d6ff-4723-ba31-890767020610]
celery.worker.job:INFO: Task pulp.server.tasks.repository.sync_with_auto_publish[24ca3aad-52b5-4eb9-a163-bc15facec00c] succeeded in 876.964413444s: <pulp.server.async.tasks.TaskResult object at 0x31df210>
celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8b06bb8f-bc50-4c54-b899-353f1328c8c2]
celery.worker.job:INFO: Task pulp.server.async.tasks._queue_release_resource[c8c20ce2-fdeb-46f3-9b5d-e991a524f610] succeeded in 0.0379554569954s: None
celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[8b06bb8f-bc50-4c54-b899-353f1328c8c2] succeeded in 0.0232481429994s: None
celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[fea8da63-dbad-401c-aa00-519d3bc54778] succeeded in 1.16808687799s: None
py.warnings:WARNING: /usr/lib/python2.7/site-packages/celery/app/control.py:34: DuplicateNodenameWarning: Received multiple replies from node name True.
py.warnings:WARNING: Please make sure you give each node a unique nodename using the `-n` option.
py.warnings:WARNING:   W_DUPNODE.format(dup),
py.warnings:WARNING:
celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[c3d6abb1-e132-407d-a61f-a3247b945ab1] succeeded in 1.29009989899s: None
celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[55ae9f20-028d-43fe-b12a-914ee50d31d3] succeeded in 1.233596609s: None
celery.worker.job:INFO: Task pulp.server.async.tasks.babysit[cdfe3d60-78c5-409e-867b-b09ee64268a6] succeeded in 1.280725091s: None
pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [rhel-6-server]
pulp.server.managers.repo.publish:ERROR: Traceback (most recent call last):
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp/server/pulp/server/managers/repo/publish.py", line 126, in _do_publish
pulp.server.managers.repo.publish:ERROR:     publish_report = publish_repo(transfer_repo, conduit, call_config)
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp/server/pulp/server/async/tasks.py", line 543, in wrap_f
pulp.server.managers.repo.publish:ERROR:     return f(*args, **kwargs)
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/distributor.py", line 130, in publish_repo
pulp.server.managers.repo.publish:ERROR:     return self._publisher.publish()
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 169, in publish
pulp.server.managers.repo.publish:ERROR:     step.process()
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 368, in process
pulp.server.managers.repo.publish:ERROR:     self._process_block()
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 603, in _process_block
pulp.server.managers.repo.publish:ERROR:     self.process_unit(package_unit)
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp_rpm/plugins/pulp_rpm/plugins/distributors/yum/publish.py", line 157, in process_unit
pulp.server.managers.repo.publish:ERROR:     self._create_symlink(source_path, destination_path)
pulp.server.managers.repo.publish:ERROR:   File "/home/rbarlow/devel/pulp/pulp/server/pulp/plugins/util/publish_step.py", line 427, in _create_symlink
pulp.server.managers.repo.publish:ERROR:     raise RuntimeError(msg % {'s': source_path})
pulp.server.managers.repo.publish:ERROR: RuntimeError: Will not create a symlink to a non-existent source [/var/lib/pulp/content/rpm/389-ds-base/1.2.10.2/15.el6/x86_64/e0c7dbd9b0d6ef7c1cee92bc3e6d8a41e5f4df3e/389-ds-base-1.2.10.2-15.el6.x86_64.rpm]

Comment 1 Chris Duryee 2014-05-22 15:45:47 UTC

I looked at this in some detail over the past few days and with bcourt. I was unable to reproduce this with the following scenario:

* install pulp 2.3
* sync a repo that contains a Packages dir[1]
* upgrade to 2.4
* publish repo

[1] note that the repo can't have a .treeinfo with repomd.xml defined right now due to #1099600

I tried some other upgrade/sync/publish combinations as well which did not cause issue. Additionally, Barnaby and I were unable to find how the scenario in comment #1 would happen during review of the code in question. Perhaps #1095829 fixed the stack you saw?

Marking as closed/worksforme, but feel free to re-open if this reappears.

Comment 3 Lukas Zapletal 2017-03-27 06:59:24 UTC

Possible steps to reproduce: Sync a repo and stop pulp services during syncing or repo generation. Then start again. The customer is doing daily full backups and syncing was still running we have found. This could be issue also with new pulp versions, could you please test this?

Comment 4 Brian Bouterse 2017-07-26 21:01:06 UTC

Pulp no longer uses this bug tracker (where Product=Pulp). I just happened to find this one in the NEW state while looking through Bugzilla. It should be in CLOSED state. I'm not sure what the right resolution is so I'm setting it back to what it was (WORKSFORME).

If you want to refile it in upstream Pulp please file here:  https://pulp.plan.io/projects/pulp/issues/new