Description of problem: I want to push an image from a remote registry to the undercloud registry but failed. Version-Release number of selected component (if applicable): OSP16 How reproducible: Steps to Reproduce: 1.run below command: sudo openstack tripleo container image push --username xxx --password xxx registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp15:15.0-1-3 2. 3. Actual results: failed with below errors: ======================= tripleo_common.image.exception.ImageUploaderException: Unable to authenticate. This may indicate missing registry credentials or the provided container or namespace does not exist. 401 Client Error: Unauthorized for url: https://registry.connect.redhat.com/auth/realms/rhc4tp/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Adellemc%2Fopenstack-cinder-volume-dellemc-rhosp15%3Apull Unable to authenticate. This may indicate missing registry credentials or the provided container or namespace does not exist. 401 Client Error: Unauthorized for url: https://registry.connect.redhat.com/auth/realms/rhc4tp/protocol/redhat-docker-v2/auth?service=docker-registry&scope=repository%3Adellemc%2Fopenstack-cinder-volume-dellemc-rhosp15%3Apull ============================= I'm sure the credentials are correct. Expected results: the command pulls the image from registry.connect.redhat.com and uploads it to lundercloud registry Additional info:
The issue is that the command only support the push destination authentication. We've added new options to provide the source auth information as well. In the mean time, the workaround would be to do a `podman pull` to fetch the container locally and then use `openstack tripleo container image push --local` to upload it.
I used 'podman pull' to fetch the container image. ============================================================ [stack@elabdir247 ~]$ podman login registry.connect.redhat.com Authenticating with existing credentials... Existing credentials are valid. Already logged in to registry.connect.redhat.com [stack@elabdir247 ~]$ podman pull registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16 Trying to pull registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16... Getting image source signatures Copying blob eae5d284042d done Copying blob ff6f434a470a done Copying blob 53723fd20694 done Copying blob 8d2d5599ee9a done Copying blob 7a53b6c263e8 done Copying blob 8aa54ab4d46b done Copying blob 0ee8d97df187 done Copying blob 4854bba0f485 done Copying blob 984abfdabe0f done Copying blob 5ad64a89996a done Copying blob ebb057a4cb92 done Copying blob 95f6dfb16592 done Copying config f9b2aa1d4e done Writing manifest to image destination Storing signatures f9b2aa1d4ecb1debeafd2624b3276efa2f67043c4c0c6551d46758eb5bf6633f [stack@elabdir247 ~]$ [stack@elabdir247 ~]$ podman images|grep registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16 registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16 latest f9b2aa1d4ecb 4 days ago 1.41 GB [stack@elabdir247 ~]$ ============================================================ But failed to upload it to local registry. ============================================================ [stack@elabdir247 ~]$ openstack tripleo container image push --local registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16 [elabdir247.ctlplane.localdomain:8787/:registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16] Failed copying the target image to the target registry [stack@elabdir247 ~]$ openstack tripleo container image push --local registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16 --debug START with options: tripleo container image push --local registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16 --debug options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interfa ce='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_ data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='', os_image_api_version='', os_key_man ager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_n ame=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='', project_domain_ name='', project_id='', project_name='', protocol='', redirect_uri='', region_name='', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', s ervice_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='', user_id='', usern ame='', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status _code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', ' object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'insp ector_api_version': '1', 'beta_command': False, 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_ap i_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []} defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'i mage_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': ' 2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'} cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_ret ries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_st ore_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'inspector_api _version': '1', 'beta_command': False, 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'w orkflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version ': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []} compute API version 2.1, cmd group openstack.compute.v2 identity API version 3, cmd group openstack.identity.v3 image API version 2, cmd group openstack.image.v2 network API version 2, cmd group openstack.network.v2 object_store API version 1, cmd group openstack.object_store.v1 volume API version 3, cmd group openstack.volume.v3 messaging API version 2, cmd group openstack.messaging.v2 database API version 1, cmd group openstack.database.v1 tripleoclient API version 1, cmd group openstack.tripleoclient.v1 data_processing API version 1.1, cmd group openstack.data_processing.v1 load_balancer API version 2.0, cmd group openstack.load_balancer.v2 neutronclient API version 2, cmd group openstack.neutronclient.v2 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 container_infra API version 1, cmd group openstack.container_infra.v1 baremetal API version 1.58, cmd group openstack.baremetal.v1 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 orchestration API version 1, cmd group openstack.orchestration.v1 dns API version 2, cmd group openstack.dns.v2 key_manager API version 1, cmd group openstack.key_manager.v1 event API version 2, cmd group openstack.event.v2 metric API version 1, cmd group openstack.metric.v1 alarming API version 2, cmd group openstack.alarming.v2 Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status _code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', ' object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'default_domain': 'default', 'timing': False, 'insp ector_api_version': '1', 'beta_command': False, 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_ap i_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []} command: tripleo container image push -> tripleoclient.v1.container_image.TripleOContainerImagePush (auth=False) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status _code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', ' object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, ' default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'beta_command': False, 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing _api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_versio n': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'region_name': '', 'auth_type': 'password', 'networks': []} run(Namespace(append_tag='', cleanup=False, dry_run=False, image_to_push='registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16', local=True, multi_arch=False, password=None, registr y_url='elabdir247.ctlplane.localdomain:8787', username=None)) take_action(Namespace(append_tag='', cleanup=False, dry_run=False, image_to_push='registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16', local=True, multi_arch=False, password=None, registry_url='elabdir247.ctlplane.localdomain:8787', username=None)) Starting new HTTPS connection (1): elabdir247.ctlplane.localdomain:8787 Starting new HTTPS connection (2): elabdir247.ctlplane.localdomain:8787 Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None) Starting new HTTP connection (1): elabdir247.ctlplane.localdomain:8787 http://elabdir247.ctlplane.localdomain:8787 "GET /v2/ HTTP/1.1" 200 2 http://elabdir247.ctlplane.localdomain:8787/v2/ status code 200 [containers-storage:registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16] Starting upload image process Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None) Starting new HTTP connection (1): elabdir247.ctlplane.localdomain:8787 http://elabdir247.ctlplane.localdomain:8787 "GET /v2/ HTTP/1.1" 200 2 http://elabdir247.ctlplane.localdomain:8787/v2/ status code 200 http://elabdir247.ctlplane.localdomain:8787 "POST /v2//blobs/uploads/ HTTP/1.1" 404 215 [elabdir247.ctlplane.localdomain:8787/:registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16] Failed copying the target image to the target registry result [] clean_up TripleOContainerImagePush: END return value: None [stack@elabdir247 ~]$ ============================================================
You need to run all those actions as root (or via sudo). [cloud-user@undercloud ~]$ sudo podman login registry.connect.redhat.com Username: xxxxxxx Password: Login Succeeded! [cloud-user@undercloud ~]$ sudo podman pull registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16 Trying to pull registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16... Getting image source signatures Copying blob ff6f434a470a skipped: already exists Copying blob eae5d284042d skipped: already exists Copying blob 7a53b6c263e8 done Copying blob 53723fd20694 done Copying blob 8d2d5599ee9a done Copying blob 8aa54ab4d46b done Copying blob 4854bba0f485 done Copying blob 0ee8d97df187 done Copying blob 5ad64a89996a done Copying blob 984abfdabe0f done Copying blob ebb057a4cb92 done Copying blob 95f6dfb16592 done Copying config f9b2aa1d4e done Writing manifest to image destination Storing signatures f9b2aa1d4ecb1debeafd2624b3276efa2f67043c4c0c6551d46758eb5bf6633f [cloud-user@undercloud ~]$ sudo openstack tripleo container image push --local registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16:latest INFO[0000] created - from /var/lib/containers/storage/overlay/f659bfaf2e3b922dcc39bb7e9011ea80508c21349fabb8d1ce2ea20525be349b/diff and /var/lib/containers/storage/overlay-layers/f659bfaf2e3b922dcc39bb7e9011ea80508c21349fabb8d1ce2ea20525be349b.tar-split.gz (wrote 20480 bytes) INFO[0008] created - from /var/lib/containers/storage/overlay/9ffebe361fec105693dc398ba724d0faa201282b8a1e9a3e6383cac48256e775/diff and /var/lib/containers/storage/overlay-layers/9ffebe361fec105693dc398ba724d0faa201282b8a1e9a3e6383cac48256e775.tar-split.gz (wrote 170629120 bytes) INFO[0015] created - from /var/lib/containers/storage/overlay/1295eae54c9d95bd8e2c7f83df2a90ac3923d89ec44231fd49f31e7a934f9656/diff and /var/lib/containers/storage/overlay-layers/1295eae54c9d95bd8e2c7f83df2a90ac3923d89ec44231fd49f31e7a934f9656.tar-split.gz (wrote 239165440 bytes) INFO[0015] created - from /var/lib/containers/storage/overlay/39231f0c35ac6b5018875a08df285edbceeeb29ab1a6185f125c5bd1d90be658/diff and /var/lib/containers/storage/overlay-layers/39231f0c35ac6b5018875a08df285edbceeeb29ab1a6185f125c5bd1d90be658.tar-split.gz (wrote 261058560 bytes) INFO[0007] created - from /var/lib/containers/storage/overlay/dd620c38e860e5ac032ef7b312c5eba3c808061a62a66894eb78fa74a80acf64/diff and /var/lib/containers/storage/overlay-layers/dd620c38e860e5ac032ef7b312c5eba3c808061a62a66894eb78fa74a80acf64.tar-split.gz (wrote 135188480 bytes) INFO[0000] created - from /var/lib/containers/storage/overlay/62d9baf75aaa3e43de193a10598b5dcdb96a064fcbfa4e5fd7860dc5f23905e2/diff and /var/lib/containers/storage/overlay-layers/62d9baf75aaa3e43de193a10598b5dcdb96a064fcbfa4e5fd7860dc5f23905e2.tar-split.gz (wrote 10397184 bytes) INFO[0000] created - from /var/lib/containers/storage/overlay/4db56ef2ef1906aa12da2d40c0b399267aa0faad2d1f44847ce338c26148fdfd/diff and /var/lib/containers/storage/overlay-layers/4db56ef2ef1906aa12da2d40c0b399267aa0faad2d1f44847ce338c26148fdfd.tar-split.gz (wrote 2560 bytes) INFO[0000] created - from /var/lib/containers/storage/overlay/e28164c097559dcd76abfe44aa0e0f99d159db34e01a38de3fd89103bae33a4e/diff and /var/lib/containers/storage/overlay-layers/e28164c097559dcd76abfe44aa0e0f99d159db34e01a38de3fd89103bae33a4e.tar-split.gz (wrote 2048 bytes) INFO[0000] created - from /var/lib/containers/storage/overlay/4af0bbebdf3b4c7f648cac91f5c666f7f6e335af674b47e8d83613170be9d51f/diff and /var/lib/containers/storage/overlay-layers/4af0bbebdf3b4c7f648cac91f5c666f7f6e335af674b47e8d83613170be9d51f.tar-split.gz (wrote 3072 bytes) INFO[0000] created - from /var/lib/containers/storage/overlay/74b22c86910f5b89ed48d4df7859da3caec4061f2db2c40b98d684ce1c41f502/diff and /var/lib/containers/storage/overlay-layers/74b22c86910f5b89ed48d4df7859da3caec4061f2db2c40b98d684ce1c41f502.tar-split.gz (wrote 7889408 bytes) INFO[0006] created - from /var/lib/containers/storage/overlay/47daaad6def148a1121ed33386516d2f79df5b9fc72b22958a06f661521a22e9/diff and /var/lib/containers/storage/overlay-layers/47daaad6def148a1121ed33386516d2f79df5b9fc72b22958a06f661521a22e9.tar-split.gz (wrote 131396096 bytes) INFO[0023] created - from /var/lib/containers/storage/overlay/dc39e39980a6a27bbbf85c1caf6297a2049551571e73e0e125f5d2ecc87a1c8b/diff and /var/lib/containers/storage/overlay-layers/dc39e39980a6a27bbbf85c1caf6297a2049551571e73e0e125f5d2ecc87a1c8b.tar-split.gz (wrote 449966080 bytes)
Hi Alex, Thanks. That works. But we have questions on how to use this build in the containers-prepare-parameter.yaml. Could you please show us how we should modify the containers-prepare-parameter.yaml to use this image for cinder-volume instead of the default image. thanks and regards Sam
So you wouldn't add this to the container-prepare-parameter.yaml. You would set ContainerCinderVolumeImage in an environment file as part of the deployment to point to your image. Explicit definition of Contianer*Image variables should override anything generated by container-prepare-parameter.yaml.
Hi Alex, Got it. That's the same we did as previous RHOSP relesaes. One more question: When will 'new options to provide the source auth information' be available? Is there any BZ for this? If yes, please add me to the cc list. Thanks and regards. Sam
Hey Sam, This bz will track the availability of the additional --source-username and --source-password options for the `openstack tripleo container image push` command.
thanks Alex.
There's another question. When we set 'ContainerCinderVolumeImage' in the environment file, should we use the 'registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16:latest' directly, or should we tag it first to local namespace and use the local namespace in the file. that is:#1 or #2? #1. ========================================== parameter_defaults: DockerCinderVolumeImage: 192.168.13.1:8787/dellemc/openstack-cinder-volume-dellemc-rhosp16:latest ========================================== #2. ========================================== parameter_defaults: DockerCinderVolumeImage: registry.connect.redhat.com/dellemc/openstack-cinder-volume-dellemc-rhosp16:latest ========================================== thanks and regards. Sam
Depends on the end user. If you're doing the local registry push, then it should be #1. If the customer is to deploy the container from the original source then you can use #2. There's also a 3rd option if the customer is using satellite server and they would use the reference to the satellite server version.
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148
*** Bug 1873967 has been marked as a duplicate of this bug. ***