Hide Forgot
Description of problem: Synchronization of a Docker-based repository fails silently in the web ui. Logs show: pulp: pulp.server.event.http:ERROR: Received HTTP 500 from HTTP notifier to https://<EDITED>/katello/api/v2/repositories/sync_complete? Maybe Dockerhub switched to v1 and we're trying to use v2 by default? Version-Release number of selected component (if applicable): Satellite 6.2.0 Compose 1 How reproducible: Steps to Reproduce: 1. Create a new Docker-based custom repository pointed at and use 'busybox' for its "Upstream Name" 2. Synchronize it via the web ui 3. Actual results: Sync process finishes successfully but no images or tags are actually downloaded Expected results: Additional info: Mar 1 07:29:21 ibm-x3250m4-01 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS Mar 1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[42465733-0d4d-42da-b48d-0596897858a6] Mar 1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[5ab2e174-b242-46f8-b67c-152f73e31e12] Mar 1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[42204355-57a7-442e-b4dd-c218efe110d1] Mar 1 07:29:21 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry.hub.docker.com Mar 1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[42465733-0d4d-42da-b48d-0596897858a6] succeeded in 0.0364322700188s: None Mar 1 07:29:21 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io Mar 1 07:29:21 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry.hub.docker.com Mar 1 07:29:21 ibm-x3250m4-01 pulp: pulp_docker.plugins.importers.sync:INFO: Could not get tags through v2 API Mar 1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:INFO: (1651-94848) {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/5ab2e174-b242-46f8-b67c-152f73e31e12/", "task_id": "5ab2e174-b242-46f8-b67c-152f73e31e12", "tags": ["pulp:repository:default_organization-docker-busybox", "pulp:action:sync"], "finish_time": null, "_ns": "task_status", "start_time": "2016-03-01T12:29:21Z", "traceback": null, "spawned_tasks": [], "progress_report": {"docker_importer": [{"num_success": 1, "description": "Downloading manifests", "step_type": "sync_step_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "8696538e-53e4-42ba-8727-037b8d7e5999", "num_processed": 1}, {"num_success": 1, "description": "Copying units already in pulp", "step_type": "get_local", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "d652e47d-be2c-4a97-86c5-8c6587d6ea3a", "num_processed": 1}, {"num_success": 1, "description": "Copying units already in pulp", "step_type": "get_local", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "6608d80f-947a-43bc-993f-64615115c2ce", "num_processed": 1}, {"num_success": 0, "description": "Downloading remote files", "step_type": "sync_step_download", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "e969ed67-1f4d-4f72-8220-d9173e4a1ca2", "num_processed": 0}, {"num_success": 0, "description": "Saving Manifests and Blobs", "step_type": "sync_step_save", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "a5e6b50c-cffa-4492-90af-e09d50840fe7", "num_processed": 0}, {"num_success": 1, "description": "Saving Tags", "step_type": "sync_step_save", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "5c944a4b-f32a-4d0a-842c-30189dd3e195", "num_process Mar 1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:INFO: (1651-94848) ed": 1}]}, "state": "running", "worker_name": "reserved_resource_worker-2@ibm-x3250m4-01.lab.eng.rdu2.redhat.com", "result": null, "error": null, "_id": {"$oid": "56d58b214666011c1f6f7b37"}, "id": "56d58b214666011c1f6f7b37"}, "event_type": "repo.sync.finish", "payload": {"importer_id": "docker_importer", "exception": null, "repo_id": "default_organization-docker-busybox", "traceback": null, "started": "2016-03-01T12:29:21Z", "_ns": "repo_sync_results", "completed": "2016-03-01T12:29:22Z", "importer_type_id": "docker_importer", "error_message": null, "summary": {"sync_step_metadata": "FINISHED", "sync_step_save": "FINISHED", "get_local": "FINISHED", "sync_step_download": "FINISHED"}, "added_count": 0, "result": "success", "updated_count": 0, "details": [{"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Downloading manifests", "num_failures": 0, "step_id": "8696538e-53e4-42ba-8727-037b8d7e5999", "step_type": "sync_step_metadata", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Copying units already in pulp", "num_failures": 0, "step_id": "d652e47d-be2c-4a97-86c5-8c6587d6ea3a", "step_type": "get_local", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Copying units already in pulp", "num_failures": 0, "step_id": "6608d80f-947a-43bc-993f-64615115c2ce", "step_type": "get_local", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Downloading remote files", "num_failures": 0, "step_id": "e969ed67-1f4d-4f72-8220-d9173e4a1ca2", "step_type": "sync_step_download", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Saving Manifests and Blobs", "num_failures": 0, "step_id": "a5e6b50c-cffa- Mar 1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:INFO: (1651-94848) 4492-90af-e09d50840fe7", "step_type": "sync_step_save", "details": ""}, {"num_processed": 1, "items_total": 0, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Saving Tags", "num_failures": 0, "step_id": "5c944a4b-f32a-4d0a-842c-30189dd3e195", "step_type": "sync_step_save", "details": ""}], "id": "56d58b224a455206731b9ab9", "removed_count": 0}} Mar 1 07:29:22 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): ibm-x3250m4-01.lab.eng.rdu2.redhat.com Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[960256b1-d9f5-44b4-bab7-83fb532b4342] Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[5ab2e174-b242-46f8-b67c-152f73e31e12] succeeded in 0.351237589988s: <pulp.server.async.tasks.TaskResult object at 0x474b090> Mar 1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:ERROR: Received HTTP 500 from HTTP notifier to https://<EDITED>/katello/api/v2/repositories/sync_complete?token=QJULQHj8reUey8QmGurksb6HUEtVjZzv. Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[42204355-57a7-442e-b4dd-c218efe110d1] succeeded in 0.00589203901473s: None Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[19c098ed-97fc-4429-a305-b45fbd8e20cb] Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[5385c0f7-9bbf-42c6-9469-442328ebab82] Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[960256b1-d9f5-44b4-bab7-83fb532b4342] succeeded in 0.0339235319989s: None Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[19c098ed-97fc-4429-a305-b45fbd8e20cb] succeeded in 0.10743384299s: {'exception': None, 'repo_id': u'default_organization-docker-busybox', 'traceback': None, 'started': '2016-03-01T12:29:22Z',... Mar 1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[5385c0f7-9bbf-42c6-9469-442328ebab82] succeeded in 0.00449386500986s: None
What feed URL and repository name were used for the sync? Here is our guidance on which docker hub URL to use for v1 vs. v2: https://github.com/pulp/pulp_docker/blob/master/docs/user-guide/recipes.rst#sync
In case we get past the originally reported problem and decide it's not a bug, I encourage you to file a separate issue about the 500 response from katello.
Michael, I tried a bunch of feed urls (as per our emails) but I also did try https://registry-1.docker.io as per your doc. Fwiw, my system has the following packages: pulp-selinux-2.8.0.0-0.5.beta.el7sat.noarch pulp-nodes-common-2.8.0.0-0.5.beta.el7sat.noarch python-pulp-puppet-common-2.8.0.0-0.5.beta.el7sat.noarch pulp-rpm-plugins-2.8.0.0-0.5.beta.el7sat.noarch python-isodate-0.5.0-4.pulp.el7sat.noarch python-pulp-oid_validation-2.8.0.0-0.5.beta.el7sat.noarch rubygem-smart_proxy_pulp-1.2.0-1.el7sat.noarch pulp-server-2.8.0.0-0.5.beta.el7sat.noarch python-pulp-streamer-2.8.0.0-0.5.beta.el7sat.noarch pulp-client-1.0-1.noarch python-pulp-common-2.8.0.0-0.5.beta.el7sat.noarch python-pulp-repoauth-2.8.0.0-0.5.beta.el7sat.noarch pulp-puppet-plugins-2.8.0.0-0.5.beta.el7sat.noarch pulp-katello-1.0-3.el7sat.noarch pulp-puppet-tools-2.8.0.0-0.5.beta.el7sat.noarch pulp-docker-plugins-2.0.0.0-0.5.beta.el7sat.noarch python-pulp-bindings-2.8.0.0-0.5.beta.el7sat.noarch python-pulp-rpm-common-2.8.0.0-0.5.beta.el7sat.noarch pulp-nodes-parent-2.8.0.0-0.5.beta.el7sat.noarch python-pulp-docker-common-2.0.0.0-0.5.beta.el7sat.noarch python-crane-0.2.2-1.el7.noarch tfm-rubygem-docker-api-1.17.0-3.el7sat.noarch tfm-rubygem-hammer_cli_foreman_docker-0.0.3-5.el7sat.noarch tfm-rubygem-foreman_docker-2.0.1-1.el7sat.noarch pulp-docker-plugins-2.0.0.0-0.5.beta.el7sat.noarch python-pulp-docker-common-2.0.0.0-0.5.beta.el7sat.noarch
Og, On my snap2 install, I am able to create and sync a docker repo with the following: name: busybox type: docker url: https://registry-1.docker.io upstream name: busybox If I provide an invalid url, I do see that the sync will finished as 'stopped'/'success'; therefore, it looks successful to the user. I will investigate that issue with this bz; however, if possible, can you retest the sunny-day case on snap2?
I have not been able to reproduce the behavior described in the initial description. In addition, I am now seeing a valid behavior when attempting the scenario mentioned in comment 5. Based on this, I am going to move the bug to ON_QA to have it looked at on the latest builds. Reference 1: Regarding comment 5: If I create the following repo: name: invalid type: docker url: https://registry-invalid.docker.io/ upstream-name: busybox Syncing the repo will result in 'stopped/warning' with Details: Could not find registry API at https://registry-invalid.docker.io/ Reference 2: The initial description did mention an error on sync_complete. That particular error occurs for syncing any repo type (e.g. yum, docker...etc). There is a separate bug to fix that error. It is bug 1317505.
VERIFIED. On Satellite 6.2 Snap4 - sync out of registry-1.docker.io works - sync out of registry.hub.docker.com doesn't While on Satellite 6.1.8 it is the opposite way. So, Satellite 6.1 is compatible with registry.hub.docker.com and Satellite 6.2 is compatible with registry-1.docker.io
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501