Bug 1313346 - Synchronization of Docker-based repositories fail silently
Synchronization of Docker-based repositories fail silently
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Content Management (Show other bugs)
6.2.0
Unspecified Unspecified
unspecified Severity high (vote)
: Beta
: --
Assigned To: Brad Buckingham
Lukas Pramuk
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-01 07:36 EST by Og Maciel
Modified: 2016-07-27 05:24 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-27 05:24:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Og Maciel 2016-03-01 07:36:01 EST
Description of problem:

Synchronization of a Docker-based repository fails silently in the web ui. Logs show:

    pulp: pulp.server.event.http:ERROR: Received HTTP 500 from HTTP notifier to https://<EDITED>/katello/api/v2/repositories/sync_complete?

Maybe Dockerhub switched to v1 and we're trying to use v2 by default?

Version-Release number of selected component (if applicable):

Satellite 6.2.0 Compose 1

How reproducible:


Steps to Reproduce:
1. Create a new Docker-based custom repository pointed at  and use 'busybox' for its "Upstream Name"
2. Synchronize it via the web ui
3.

Actual results:

Sync process finishes successfully but no images or tags are actually downloaded

Expected results:


Additional info:

Mar  1 07:29:21 ibm-x3250m4-01 pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Mar  1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[42465733-0d4d-42da-b48d-0596897858a6]
Mar  1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[5ab2e174-b242-46f8-b67c-152f73e31e12]
Mar  1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[42204355-57a7-442e-b4dd-c218efe110d1]
Mar  1 07:29:21 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): registry.hub.docker.com
Mar  1 07:29:21 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[42465733-0d4d-42da-b48d-0596897858a6] succeeded in 0.0364322700188s: None
Mar  1 07:29:21 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): auth.docker.io
Mar  1 07:29:21 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): registry.hub.docker.com
Mar  1 07:29:21 ibm-x3250m4-01 pulp: pulp_docker.plugins.importers.sync:INFO: Could not get tags through v2 API
Mar  1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:INFO: (1651-94848) {"call_report": {"exception": null, "task_type": "pulp.server.managers.repo.sync.sync", "_href": "/pulp/api/v2/tasks/5ab2e174-b242-46f8-b67c-152f73e31e12/", "task_id": "5ab2e174-b242-46f8-b67c-152f73e31e12", "tags": ["pulp:repository:default_organization-docker-busybox", "pulp:action:sync"], "finish_time": null, "_ns": "task_status", "start_time": "2016-03-01T12:29:21Z", "traceback": null, "spawned_tasks": [], "progress_report": {"docker_importer": [{"num_success": 1, "description": "Downloading manifests", "step_type": "sync_step_metadata", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "8696538e-53e4-42ba-8727-037b8d7e5999", "num_processed": 1}, {"num_success": 1, "description": "Copying units already in pulp", "step_type": "get_local", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "d652e47d-be2c-4a97-86c5-8c6587d6ea3a", "num_processed": 1}, {"num_success": 1, "description": "Copying units already in pulp", "step_type": "get_local", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "6608d80f-947a-43bc-993f-64615115c2ce", "num_processed": 1}, {"num_success": 0, "description": "Downloading remote files", "step_type": "sync_step_download", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "e969ed67-1f4d-4f72-8220-d9173e4a1ca2", "num_processed": 0}, {"num_success": 0, "description": "Saving Manifests and Blobs", "step_type": "sync_step_save", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "a5e6b50c-cffa-4492-90af-e09d50840fe7", "num_processed": 0}, {"num_success": 1, "description": "Saving Tags", "step_type": "sync_step_save", "items_total": 0, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "5c944a4b-f32a-4d0a-842c-30189dd3e195", "num_process
Mar  1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:INFO: (1651-94848) ed": 1}]}, "state": "running", "worker_name": "reserved_resource_worker-2@ibm-x3250m4-01.lab.eng.rdu2.redhat.com", "result": null, "error": null, "_id": {"$oid": "56d58b214666011c1f6f7b37"}, "id": "56d58b214666011c1f6f7b37"}, "event_type": "repo.sync.finish", "payload": {"importer_id": "docker_importer", "exception": null, "repo_id": "default_organization-docker-busybox", "traceback": null, "started": "2016-03-01T12:29:21Z", "_ns": "repo_sync_results", "completed": "2016-03-01T12:29:22Z", "importer_type_id": "docker_importer", "error_message": null, "summary": {"sync_step_metadata": "FINISHED", "sync_step_save": "FINISHED", "get_local": "FINISHED", "sync_step_download": "FINISHED"}, "added_count": 0, "result": "success", "updated_count": 0, "details": [{"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Downloading manifests", "num_failures": 0, "step_id": "8696538e-53e4-42ba-8727-037b8d7e5999", "step_type": "sync_step_metadata", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Copying units already in pulp", "num_failures": 0, "step_id": "d652e47d-be2c-4a97-86c5-8c6587d6ea3a", "step_type": "get_local", "details": ""}, {"num_processed": 1, "items_total": 1, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Copying units already in pulp", "num_failures": 0, "step_id": "6608d80f-947a-43bc-993f-64615115c2ce", "step_type": "get_local", "details": ""}, {"num_processed": 0, "items_total": 0, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Downloading remote files", "num_failures": 0, "step_id": "e969ed67-1f4d-4f72-8220-d9173e4a1ca2", "step_type": "sync_step_download", "details": ""}, {"num_processed": 0, "items_total": 1, "state": "FINISHED", "num_success": 0, "error_details": [], "description": "Saving Manifests and Blobs", "num_failures": 0, "step_id": "a5e6b50c-cffa-
Mar  1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:INFO: (1651-94848) 4492-90af-e09d50840fe7", "step_type": "sync_step_save", "details": ""}, {"num_processed": 1, "items_total": 0, "state": "FINISHED", "num_success": 1, "error_details": [], "description": "Saving Tags", "num_failures": 0, "step_id": "5c944a4b-f32a-4d0a-842c-30189dd3e195", "step_type": "sync_step_save", "details": ""}], "id": "56d58b224a455206731b9ab9", "removed_count": 0}}
Mar  1 07:29:22 ibm-x3250m4-01 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): ibm-x3250m4-01.lab.eng.rdu2.redhat.com
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[960256b1-d9f5-44b4-bab7-83fb532b4342]
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[5ab2e174-b242-46f8-b67c-152f73e31e12] succeeded in 0.351237589988s: <pulp.server.async.tasks.TaskResult object at 0x474b090>
Mar  1 07:29:22 ibm-x3250m4-01 pulp: pulp.server.event.http:ERROR: Received HTTP 500 from HTTP notifier to https://<EDITED>/katello/api/v2/repositories/sync_complete?token=QJULQHj8reUey8QmGurksb6HUEtVjZzv.
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[42204355-57a7-442e-b4dd-c218efe110d1] succeeded in 0.00589203901473s: None
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[19c098ed-97fc-4429-a305-b45fbd8e20cb]
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[5385c0f7-9bbf-42c6-9469-442328ebab82]
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[960256b1-d9f5-44b4-bab7-83fb532b4342] succeeded in 0.0339235319989s: None
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.publish.publish[19c098ed-97fc-4429-a305-b45fbd8e20cb] succeeded in 0.10743384299s: {'exception': None, 'repo_id': u'default_organization-docker-busybox', 'traceback': None, 'started': '2016-03-01T12:29:22Z',...
Mar  1 07:29:22 ibm-x3250m4-01 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[5385c0f7-9bbf-42c6-9469-442328ebab82] succeeded in 0.00449386500986s: None
Comment 2 Michael Hrivnak 2016-03-01 09:22:34 EST
What feed URL and repository name were used for the sync?

Here is our guidance on which docker hub URL to use for v1 vs. v2: https://github.com/pulp/pulp_docker/blob/master/docs/user-guide/recipes.rst#sync
Comment 3 Michael Hrivnak 2016-03-01 13:06:31 EST
In case we get past the originally reported problem and decide it's not a bug, I encourage you to file a separate issue about the 500 response from katello.
Comment 4 Og Maciel 2016-03-01 17:37:06 EST
Michael,

I tried a bunch of feed urls (as per our emails) but I also did try https://registry-1.docker.io as per your doc.

Fwiw, my system has the following packages:

pulp-selinux-2.8.0.0-0.5.beta.el7sat.noarch
pulp-nodes-common-2.8.0.0-0.5.beta.el7sat.noarch
python-pulp-puppet-common-2.8.0.0-0.5.beta.el7sat.noarch
pulp-rpm-plugins-2.8.0.0-0.5.beta.el7sat.noarch
python-isodate-0.5.0-4.pulp.el7sat.noarch
python-pulp-oid_validation-2.8.0.0-0.5.beta.el7sat.noarch
rubygem-smart_proxy_pulp-1.2.0-1.el7sat.noarch
pulp-server-2.8.0.0-0.5.beta.el7sat.noarch
python-pulp-streamer-2.8.0.0-0.5.beta.el7sat.noarch
pulp-client-1.0-1.noarch
python-pulp-common-2.8.0.0-0.5.beta.el7sat.noarch
python-pulp-repoauth-2.8.0.0-0.5.beta.el7sat.noarch
pulp-puppet-plugins-2.8.0.0-0.5.beta.el7sat.noarch
pulp-katello-1.0-3.el7sat.noarch
pulp-puppet-tools-2.8.0.0-0.5.beta.el7sat.noarch
pulp-docker-plugins-2.0.0.0-0.5.beta.el7sat.noarch
python-pulp-bindings-2.8.0.0-0.5.beta.el7sat.noarch
python-pulp-rpm-common-2.8.0.0-0.5.beta.el7sat.noarch
pulp-nodes-parent-2.8.0.0-0.5.beta.el7sat.noarch
python-pulp-docker-common-2.0.0.0-0.5.beta.el7sat.noarch
python-crane-0.2.2-1.el7.noarch
tfm-rubygem-docker-api-1.17.0-3.el7sat.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.3-5.el7sat.noarch
tfm-rubygem-foreman_docker-2.0.1-1.el7sat.noarch
pulp-docker-plugins-2.0.0.0-0.5.beta.el7sat.noarch
python-pulp-docker-common-2.0.0.0-0.5.beta.el7sat.noarch
Comment 5 Brad Buckingham 2016-03-02 15:04:27 EST
Og, 

On my snap2 install, I am able to create and sync a docker repo with the following: 

name: busybox
type: docker
url: https://registry-1.docker.io
upstream name: busybox

If I provide an invalid url, I do see that the sync will finished as 'stopped'/'success'; therefore, it looks successful to the user.  I will investigate that issue with this bz; however, if possible, can you retest the sunny-day case on snap2?
Comment 6 Brad Buckingham 2016-03-17 16:12:21 EDT
I have not been able to reproduce the behavior described in the initial description.  In addition, I am now seeing a valid behavior when attempting the scenario mentioned in comment 5.  Based on this, I am going to move the bug to ON_QA to have it looked at on the latest builds.

Reference 1:

Regarding comment 5: If I create the following repo:
   name: invalid
   type: docker
   url: https://registry-invalid.docker.io/
   upstream-name: busybox

Syncing the repo will result in 'stopped/warning' with Details:
  Could not find registry API at https://registry-invalid.docker.io/


Reference 2:

The initial description did mention an error on sync_complete.  That particular error occurs for syncing any repo type (e.g. yum, docker...etc).  There is a separate bug to fix that error.  It is bug 1317505.
Comment 7 Lukas Pramuk 2016-03-17 18:18:13 EDT
VERIFIED.

On Satellite 6.2 Snap4 
- sync out of registry-1.docker.io works
- sync out of registry.hub.docker.com doesn't
While on Satellite 6.1.8 it is the opposite way. 
 
So, Satellite 6.1 is compatible with registry.hub.docker.com
and Satellite 6.2 is compatible with registry-1.docker.io
Comment 10 errata-xmlrpc 2016-07-27 05:24:15 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501

Note You need to log in before you can comment on or make changes to this bug.