Description of problem: synchronization of repositories from Red Hat CDN fails on the publish step. Version-Release number of selected component (if applicable): # rpm -qa | grep pulp python-pulp-common-2.4.0-0.10.beta.el6.noarch createrepo-0.9.9-21.2.pulp.el6.noarch mod_wsgi-3.4-1.pulp.el6.x86_64 pulp-server-2.4.0-0.10.beta.el6.noarch pulp-selinux-2.4.0-0.10.beta.el6.noarch pulp-admin-client-2.4.0-0.10.beta.el6.noarch python-isodate-0.5.0-1.pulp.el6.noarch python-pulp-client-lib-2.4.0-0.10.beta.el6.noarch m2crypto-0.21.1.pulp-8.el6.x86_64 python-pulp-puppet-common-2.4.0-0.10.beta.el6.noarch pulp-puppet-plugins-2.4.0-0.10.beta.el6.noarch pulp-rpm-admin-extensions-2.4.0-0.10.beta.el6.noarch python-pulp-rpm-common-2.4.0-0.10.beta.el6.noarch python-kombu-3.0.15-4.pulp.el6.noarch pulp-rpm-plugins-2.4.0-0.10.beta.el6.noarch python-pulp-bindings-2.4.0-0.10.beta.el6.noarch pulp-puppet-admin-extensions-2.4.0-0.10.beta.el6.noarch How reproducible: Always Steps to Reproduce: 1. Create repository # pulp-admin rpm repo create --repo-id=live-rhel-6-server-x86_64-rhscl --relative-url=live-rhel-6-server-x86_64-rhscl --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/rhscl/1/os --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem --feed-key=/etc/pki/entitlement/<some-key>-key.pem --feed-cert=/etc/pki/entitlement/<some-cert>.pem --serve-http=true --serve-https=true 2. Sync the repo # # pulp-admin rpm repo sync run --repo-id live-rhel-6-server-x86_64-rhscl +----------------------------------------------------------------------+ Synchronizing Repository [live-rhel-6-server-x86_64-rhscl] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [-] ... completed Downloading repository content... [==================================================] 100% RPMs: 0/0 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 0/0 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [-] ... completed Task Succeeded Publishing Distribution files [-] ... completed Publishing RPMs [ ] 0% 0 of 525 items Task Failed Pulp exception occurred: PulpExecutionException 3. Actual results: The sync operation fails during the publish steps, below is from the log files: May 2 16:04:47 pulp-server-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[a3690f2b-6a05-4a3d-8c02-1576ecbfebdf] May 2 16:04:47 pulp-server-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[a3690f2b-6a05-4a3d-8c02-1576ecbfebdf] succeeded in 0.016254301001s: u'reserved_resource_worker-4.com' May 2 16:04:48 pulp-server-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.tasks.repository.sync_with_auto_publish[b6637cb5-bbdc-4af5-bb53-666b8f3ca249] May 2 16:04:48 pulp-server-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_release_resource[3dd92c1e-e6dc-4fe8-b8e3-7d190a2efefc] May 2 16:04:48 pulp-server-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 2 16:04:49 pulp-server-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 2 16:04:49 pulp-server-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 2 16:04:49 pulp-server-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 2 16:04:49 pulp-server-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 2 16:05:19 pulp-server-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 2 16:05:20 pulp-server-01 pulp: nectar.downloaders.threaded:ERROR: Download of https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/rhscl/1/os/.treeinfo failed with code 404: Not Found May 2 16:05:21 pulp-server-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com May 2 16:05:23 pulp-server-01 pulp: nectar.downloaders.threaded:ERROR: Download of https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/rhscl/1/os/treeinfo failed with code 404: Not Found May 2 16:05:24 pulp-server-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._reserve_resource[d873f1e3-5380-4efb-aaab-f1135a305635] May 2 16:05:24 pulp-server-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._reserve_resource[d873f1e3-5380-4efb-aaab-f1135a305635] succeeded in 0.0123427050021s: u'reserved_resource_worker-4.com' May 2 16:05:24 pulp-server-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[425d2c3b-551c-4eac-b5f4-0e0bcf673f04] May 2 16:05:24 pulp-server-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_release_resource[5ed71ae2-c4a1-425c-8926-c85310be8c05] May 2 16:05:24 pulp-server-01 pulp: celery.worker.job:INFO: Task pulp.server.tasks.repository.sync_with_auto_publish[b6637cb5-bbdc-4af5-bb53-666b8f3ca249] succeeded in 36.766929609s: <pulp.server.async.tasks.TaskResult object at 0x2473c50> May 2 16:05:24 pulp-server-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[6715b215-0c08-45c9-b397-dc519c76b742] May 2 16:05:24 pulp-server-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_release_resource[3dd92c1e-e6dc-4fe8-b8e3-7d190a2efefc] succeeded in 0.0337830839999s: None May 2 16:05:24 pulp-server-01 rsyslogd-2177: imuxsock begins to drop messages from pid 3800 due to rate-limiting May 2 16:05:24 pulp-server-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[6715b215-0c08-45c9-b397-dc519c76b742] succeeded in 0.0431937829999s: None May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: Exception caught from plugin during publish for repo [live-rhel-6-server-x86_64-rhscl] May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: Traceback (most recent call last): May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/publish.py", line 126, in _do_publish May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: publish_report = publish_repo(transfer_repo, conduit, call_config) May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 563, in wrap_f May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: return f(*args, **kwargs) May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 128, in publish_repo May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: return self._publisher.publish() May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 169, in publish May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: step.process() May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 368, in process May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: self._process_block() May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 603, in _process_block May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: self.process_unit(package_unit) May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 163, in process_unit May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: self._create_symlink(source_path, destination_path) May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 427, in _create_symlink May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: raise RuntimeError(msg % {'s': source_path}) May 2 16:05:25 pulp-server-01 pulp: pulp.server.managers.repo.publish:ERROR: RuntimeError: Will not create a symlink to a non-existent source [/var/lib/pulp/content/rpm/mariadb55/1/7.el6/x86_64/99b3c45942fe34c45fe0df7c18732dc62b107e6a/Packages/mariadb55-1-7.el6.x86_64.rpm] It complains about an non existing source for mariadb55-1-7.el6.x86_64.rpm. If i search for the rpm in the repo i can find it. # pulp-admin rpm repo content rpm --str-eq="filename=mariadb55-1-7.el6.x86_64.rpm" --repo-id live-rhel-6-server-x86_64-rhscl Arch: x86_64 Buildhost: x86-022.build.eng.bos.redhat.com Checksum: 99b3c45942fe34c45fe0df7c18732dc62b107e6a Checksumtype: sha Description: This is the main package for mariadb55 Software Collection, which installs necessary packages to use MariaDB 5.5 server, a community developed branch of MySQL. Software Collections allow to install more versions of the same package by using alternative directory structure. Install this package if you want to use MariaDB 5.5 server on your system. Epoch: 0 Filename: mariadb55-1-7.el6.x86_64.rpm License: GPLv2+ Name: mariadb55 Provides: mariadb55 = 1-7.el6-0, mariadb55(x86-64) = 1-7.el6-0 Release: 7.el6 Requires: mariadb55-mariadb-server, scl-utils Vendor: None Version: 1 If i do an locate on the rpm i can see that it's stored under /var/lib/pulp/content/rpm/mariadb55/1/7.el6/x86_64/99b3c45942fe34c45fe0df7c18732dc62b107e6a/ and not under /var/lib/pulp/content/rpm/mariadb55/1/7.el6/x86_64/99b3c45942fe34c45fe0df7c18732dc62b107e6a/Packages/ where the publish operation is looking. # locate mariadb55-1-7.el6.x86_64.rpm /var/lib/pulp/content/rpm/mariadb55/1/7.el6/x86_64/99b3c45942fe34c45fe0df7c18732dc62b107e6a/mariadb55-1-7.el6.x86_64.rpm So either the rpm is stored in the wrong location or the publish operation is looking at wrong place. Trying to sync another repo ends up with same result except it complains on different rpms. Expected results: Additional info:
This problem occurs when the repos I'm trying to sync from CDN contains rpms that exist in repos that i have synced from another pulp(2.3) server. for example: If i first sync an cloned repo from another pulp server I can see that the rpms in that repo has below storage_path: u'repository_memberships': [u'cloned-rhel-6-server-x86_64-rhscl'] u'_storage_path': u'/var/lib/pulp/content/rpm/mariadb55/1/7.el6/x86_64/99b3c45942fe34c45fe0df7c18732dc62b107e6a/mariadb55-1-7.el6.x86_64.rpm' If i then sync an repo from CDN that contains that rpm, the storage path for the rpm is changed to: u'repository_memberships': [u'cloned-rhel-6-server-x86_64-rhscl', u'live-rhel-6-server-x86_64-rhscl'] u'_storage_path': u'/var/lib/pulp/content/rpm/mariadb55/1/7.el6/x86_64/99b3c45942fe34c45fe0df7c18732dc62b107e6a/Packages/mariadb55-1-7.el6.x86_64.rpm (Packages) is added to the storage path for the rpm but the rpm is stored in the old storage path. So the publish fails due to it looking for the rpm in an storage path where it's not exist. I can successfully sync repos from CDN that not contains rpms that exist in another repo on the pulp server.
https://github.com/pulp/pulp_rpm/pull/483
(In reply to Chris Duryee from comment #2) > https://github.com/pulp/pulp_rpm/pull/483 thanks, I have verified it in my setup and it works.
The fix for this bug is included in the pulp-2.4.0-0.14.beta builds.
Created and synced 2 fedora repos that have the same feed but different relative-url. In both cases publish succeeded and the storage_path is the same. # pulp-admin -u admin -p admin rpm repo create --repo-id f20 --feed http://ftp.linux.cz/pub/linux/fedora/linux/releases/20/Fedora/x86_64/os/ Successfully created repository [f20] [root@ec2-54-74-97-52 ~]# pulp-admin -u admin -p admin rpm repo sync run --repo-id f20 +----------------------------------------------------------------------+ Synchronizing Repository [f20] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [-] ... completed Downloading repository content... [==================================================] 100% RPMs: 4071/4071 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 7/7 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [|] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 4071 of 4071 items ... completed Publishing Delta RPMs [-] ... completed Publishing Errata [-] ... completed Publishing Comps file [==================================================] 100% 124 of 124 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Publishing files to web [\] ... completed Writing Listings File [-] ... completed Task Succeeded [root@ec2-54-74-97-52 ~]# ll /var/lib/pulp/content/rpm/zsh/5.0.2/6.fc20/x86_64/b4c4de02d821b7dd39ea31f81a25045995f02d26423a3c54e79fda4e66c35c54/zsh-5.0.2-6.fc20.x86_64.rpm -rw-r--r--. 1 apache apache 2471656 Jun 6 12:07 /var/lib/pulp/content/rpm/zsh/5.0.2/6.fc20/x86_64/b4c4de02d821b7dd39ea31f81a25045995f02d26423a3c54e79fda4e66c35c54/zsh-5.0.2-6.fc20.x86_64.rpm [root@ec2-54-74-97-52 ~]# pulp-admin -u admin -p admin rpm repo create --repo-id f20.1 --feed http://ftp.linux.cz/pub/linux/fedora/linux/releases/20/Fedora/x86_64/os/ --relative-url myf20 Successfully created repository [f20.1] [root@ec2-54-74-97-52 ~]# pulp-admin -u admin -p admin rpm repo sync run --repo-id f20.1 +----------------------------------------------------------------------+ Synchronizing Repository [f20.1] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [/] ... completed Downloading repository content... [==================================================] 100% RPMs: 0/0 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 7/7 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [|] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 4071 of 4071 items ... completed Publishing Delta RPMs [-] ... completed Publishing Errata [-] ... completed Publishing Comps file [==================================================] 100% 124 of 124 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Publishing files to web [\] ... completed Writing Listings File [-] ... completed Task Succeeded [root@ec2-54-74-97-52 ~]# ll /var/lib/pulp/content/rpm/zsh/5.0.2/6.fc20/x86_64/b4c4de02d821b7dd39ea31f81a25045995f02d26423a3c54e79fda4e66c35c54/zsh-5.0.2-6.fc20.x86_64.rpm -rw-r--r--. 1 apache apache 2471656 Jun 6 12:07 /var/lib/pulp/content/rpm/zsh/5.0.2/6.fc20/x86_64/b4c4de02d821b7dd39ea31f81a25045995f02d26423a3c54e79fda4e66c35c54/zsh-5.0.2-6.fc20.x86_64.rpm Tested in 2.4.0-0.19.beta Moving to Verified.
Moving this back to on-qa since the bug is specifically talking about CDN syncs. And the verification seems to have been done with f20 feed
Preethi, I've talked to Chris before verifying the bug and he confirmed that it does not matter whether sync is from CDN or not, anything with Packages directory would fit. But thank you for your input - really appreciated :)
This has been fixed in Pulp 2.4.0-1.
*** Bug 1025903 has been marked as a duplicate of this bug. ***