Description of problem: When you have a repo with proxy, repo sync seems to be failing Version-Release number of selected component (if applicable): pulp-server-2.5.0-0.16.rc.el7.noarch How reproducible: Steps to Reproduce: 1.create a repo with proxy 2. Repo sync 3. Actual results: [root@cloud-qe-19 ~]# pulp-admin rpm repo create --repo-id pulp-unittest --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --proxy-host http://pulp-proxy.usersys.redhat.com --proxy-port 8080 --proxy-user admin --proxy-pass admin Successfully created repository [pulp-unittest] [root@cloud-qe-19 ~]# [root@cloud-qe-19 ~]# [root@cloud-qe-19 ~]# pulp-admin rpm repo sync run --repo-id pulp-unittest+----------------------------------------------------------------------+ Synchronizing Repository [pulp-unittest] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [\] ... failed [Errno 2] No such file or directory: u'/var/lib/pulp/working/repos/pulp-unittest/importers/yum_importer/tmpkROymL/oth er.xml.gz' Task Failed Importer indicated a failed respons Expected results: Additional info: Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: Traceback (most recent call last): Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 104, in run Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files = self.get_metadata() Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 211, in get_metadata Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: metadata_files.generate_dbs() Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py", line 267, in generate_dbs Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: xml_file_handle = self.get_metadata_file_handle(filename) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/metadata.py", line 238, in get_metadata_file_handle Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: file_handle = gzip.open(file_path, 'r') Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib64/python2.7/gzip.py", line 34, in open Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: return GzipFile(filename, mode, compresslevel) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: File "/usr/lib64/python2.7/gzip.py", line 94, in __init__ Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb') Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27989]: pulp_rpm.plugins.importers.yum.sync:ERROR: IOError: [Errno 2] No such file or directory: u'/var/lib/pulp/working/repos/pulp-unittest/importers/yum_importer/tmpkROymL/other.xml.gz' Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: Task pulp.server.tasks.repository.sync_with_auto_publish[0a8d71ea-48a9-48e9-9c4e-dc3ba7ecc898] raised unexpected: PulpExecutionException('Importer indicated a failed response',) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: Traceback (most recent call last): Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: R = retval = fun(*args, **kwargs) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 315, in __call__ Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: return super(Task, self).__call__(*args, **kwargs) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: return self.run(*args, **kwargs) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/tasks/repository.py", line 210, in sync_with_auto_publish Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: sync_result = managers.repo_sync_manager().sync(repo_id, sync_config_override=overrides) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 113, in sync Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: raise PulpExecutionException(_('Importer indicated a failed response')) Nov 05 11:06:57 cloud-qe-19.idmqe.lab.eng.bos.redhat.com pulp[27918]: celery.worker.job:ERROR: PulpExecutionException: Importer indicated a failed response
adding '--max-downloads 15' will work around this issue. There appears to be some problem when the same downloader is used more than once.
Can you retry using a different ssl-enabled repo? https://mirror.umd.edu/fedora/linux/releases/20/Fedora/x86_64/os/ worked ok for me
I suspect something may be going on with repos.fedorapeople.org. I was unable to sync the pulp_unittest repo, the pulp repo, or the katello repo via a proxy. However I was able to sync fedora 20 from the URL above ok, and was also able to sync from the CDN via a proxy. Additionally, I was able to repro the behavior outlined in comment #0 in both pulp 2.5 and 2.4.0 (which is known to work in the past with fedorapeople.org). I used both tinyproxy and squid and did not notice any behavior difference between the two. I see some stuff that might or might not be unusual in the tcpdump output when the proxy connects to repos.fedorapeople.org. I'll find another place to host the pulp_unittest repo tomorrow and we'll see if it works from there. If it does I'll follow up with the fedora infra team to see if something is different about their servers.
I was able to sync from this repo without issue via a proxy: https://cduryee-repotest.rhcloud.com/static/pulp_unittest/ I think something is different (possibly recently changed) about repos.fedorapeople.org that is causing the issue. It is certainly worth investigation and may possibly be a Pulp bug since I can't repro with curl, but I don't think it should block 2.5.0. Is it ok to unassign the target release from this bz?
moving off 2.5.0 per mhrivnak
update: I am able to repro this behavior with a small script using python-requests 2.2.1, but not 2.4.3. I suspect this is a bug that got fixed since 2.2.1. I did a "pip install --upgrade requests" and Pulp is able to sync the repo again. I think upgrading the dep to 2.4.3 will fix this bz. Moving to 2.5.1.
https://github.com/pulp/pulp/pull/1290 https://github.com/pulp/nectar/pull/25
pulp/deps change merged to 2.5-dev and master. I did not merge the nectar change after discussion with rbarlow.
fixed in pulp 2.6.0-0.2.beta
[root@cloud-qe-4 ~]# rpm -qa python-requests python-requests-2.4.3-1.el7.noarch [root@cloud-qe-4 ~]# rpm -qa pulp-server pulp-server-2.6.0-0.5.beta.el7.noarch [root@cloud-qe-4 ~]# [root@cloud-qe-4 ~]# pulp-admin rpm repo create --repo-id pulp-unittest --feed https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/pulp_unittest/ --proxy-host http://ibm-x3550m3-11.lab.eng.brq.redhat.com --proxy-port 3128 Successfully created repository [pulp-unittest] [root@cloud-qe-4 ~]# [root@cloud-qe-4 ~]# [root@cloud-qe-4 ~]# pulp-admin rpm repo sync run --repo-id pulp-unittest +----------------------------------------------------------------------+ Synchronizing Repository [pulp-unittest] +----------------------------------------------------------------------+ This command may be exited via ctrl+c without affecting the request. Downloading metadata... [\] ... completed Downloading repository content... [==================================================] 100% RPMs: 3/3 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 3/3 items ... completed Importing errata... [\] ... completed Importing package groups/categories... [-] ... completed Task Succeeded Initializing repo metadata [-] ... completed Publishing Distribution files [-] ... completed Publishing RPMs [==================================================] 100% 3 of 3 items ... completed Publishing Delta RPMs ... skipped Publishing Errata [==================================================] 100% 52 of 52 items ... completed Publishing Comps file [==================================================] 100% 5 of 5 items ... completed Publishing Metadata. [-] ... completed Closing repo metadata [-] ... completed Generating sqlite files ... skipped Publishing files to web [-] ... completed Writing Listings File [-] ... completed Task Succeeded
(In reply to Chris Duryee from comment #1) > adding '--max-downloads 15' will work around this issue. There appears to be > some problem when the same downloader is used more than once. Assuming I want to create a repository via hammer or WebUI for Sat6, what (other) pulp-admin options to use for the workaround? Here is the list of options: # pulp-admin rpm repo create --help | awk '{ print $1 }' | grep "^--" --repo-id --display-name --description --note --feed --validate --skip --feed-ca-cert --verify-feed-ssl --feed-cert --feed-key --proxy-host --proxy-port --proxy-user --proxy-pass --max-downloads --max-speed --remove-missing --retain-old-count --relative-url --serve-http --serve-https --checksum-type --gpg-key --generate-sqlite --host-ca --auth-ca --auth-cert #
Pavel, I would confirm first that this is indeed the issue since the stack trace it gives is a bit generic. There is a small script in the commit message for https://github.com/pulp/pulp/pull/1290 that you can use. If you update the proxy_url and base_url and then run the script on the machine in question, you can see if this BZ is indeed the issue. I am not sure how to set --max-downloads via hammer or the web UI. We can investigate that but it may be easier to just update the nectar RPM on the system.
Moved to https://pulp.plan.io/issues/609