Description of problem: OSP13 is failing during containers preparation while trying to push containers on the local registry: clean_up UploadImage: Could not push image 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 64, in take_action uploader.upload() File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 96, in upload uploader.run_tasks() File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 388, in run_tasks result = self.upload_image(*first) File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 184, in upload_image DockerImageUploader._push(dockerc, new_repo, tag=tag) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 214, in wrapped_f return self.call(f, *args, **kw) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call start_time=start_time) File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 265, in iter raise RetryError(fut).reraise() File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 344, in reraise raise self.last_attempt.result() File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result return self.__get_result() File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call result = fn(*args, **kwargs) File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 217, in _push raise ImageUploaderException('Could not push image %s' % image) ImageUploaderException: Could not push image 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server The command used to complete the action is: openstack overcloud container image upload --debug --config-file /home/stack/overcloud_containers.yaml this command was used to generate overcloud_containers.yaml file: openstack overcloud container image prepare \ --output-images-file /home/stack/overcloud_containers.yaml \ --output-env-file /home/stack/containers-default-parameters.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ --namespace docker-registry.engineering.redhat.com/rhosp13 \ --tag $BUILD_ID \ --prefix "openstack" \ --suffix "" \ --set "ceph_namespace=docker.io/ceph" \ --set "ceph_image=daemon" \ --set "ceph_tag=v3.1.0-stable-3.1-luminous-centos-7-x86_64" \ --push-destination 192.168.24.1:8787 The docker-registry.engineering.redhat.com is configured as INSECURE_REGISTRY inside /etc/sysconfig/docker. Version-Release number of selected component (if applicable): Puddle 2018-10-02.1 How reproducible: The failure is consistent. Steps to Reproduce: 1. Just generate the overcloud_containers.yaml file and launch the above openstack overcloud container image upload command Actual results: Could not push any image. Expected results: All the images pushed correctly. Additional info: sosreports on their way.
Can you please provide the following: - generated overcloud_containers.yaml - content of /etc/docker-distribution/registry/config.yml - output of: sudo systemctl status docker-distribution
Hi Steve, here's what you requested: [stack@undercloud ~]$ cat overcloud_containers.yaml container_images: - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-evaluator:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-listener:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-notifier:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-central:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-compute:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-notification:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-scheduler:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-volume:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cron:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-glance-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-metricd:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-statsd:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-haproxy:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-api-cfn:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-engine:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-horizon:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-iscsid:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-keystone:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-mariadb:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-memcached:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-dhcp-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-l3-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-metadata-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-openvswitch-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-server:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-compute:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-conductor:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-consoleauth:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-libvirt:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-novncproxy:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-placement-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-scheduler:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-panko-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-rabbitmq:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-redis:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-account:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-container:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-object:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-proxy-server:2018-10-02.1 push_destination: 192.168.24.1:878 [stack@undercloud ~]$ cat /etc/docker-distribution/registry/config.yml version: 0.1 log: fields: service: registry storage: cache: layerinfo: inmemory filesystem: rootdirectory: /var/lib/registry http: addr: 192.168.24.1:8787 [stack@undercloud ~]$ sudo systemctl status docker-distribution ● docker-distribution.service - v2 Registry server for Docker Loaded: loaded (/usr/lib/systemd/system/docker-distribution.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-10-22 07:23:18 EDT; 19h ago Main PID: 17684 (registry) Tasks: 13 CGroup: /system.slice/docker-distribution.service └─17684 /usr/bin/registry serve /etc/docker-distribution/registry/config.yml Oct 23 03:22:10 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:10 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:12 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:12 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:14 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:14 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:16 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:16 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:18 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:18 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:20 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:20 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:22 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:22 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:24 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:24 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:26 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:26 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:28 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:28 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
192.168.24.1:8787 should also be an insecure registry in /etc/sysconfig/docker. To do this the undercloud.conf needs the following before the undercloud is installed: docker_insecure_registries = 192.168.24.1:8787, docker-registry.engineering.redhat.com To unblock the production chain, this is what I suggest, but we need to decide what to do for the customer. OSP-14 adds 192.168.24.1:8787 automatically, but instack based OSP-13 doesn't. We can do one of the following: - Document that they must always manually add docker_insecure_registries = 192.168.24.1:8787 to undercloud.conf - Propose a fix to instack-undercloud to append local_ip:8787 DOCKER_INSECURE_REGISTRIES if it isn't there
Hey all, I'm still hitting this on the latest Puddle (2018-11-07.3) with package "instack-undercloud-8.4.6-2.el7ost.noarch", so not yet the one mentioned in "Fixed in version". What else needs to be done to make this move?
[stack@undercloud-0 ~]$ cat undercloud.conf [DEFAULT] # Network interface on the Undercloud that will be handling the PXE # boots and DHCP for Overcloud instances. (string value) local_interface = eth0 local_ip = 192.168.0.1/24 undercloud_public_host = 192.168.0.2 undercloud_admin_host = 192.168.0.3 #TODO: use release >= 10 when RHBZ#1633193 is resolved undercloud_ntp_servers=clock.redhat.com hieradata_override = /home/stack/hiera_override.yaml [ctlplane-subnet] local_subnet = ctlplane-subnet cidr = 192.168.0.0/24 dhcp_start = 192.168.0.5 dhcp_end = 192.168.0.24 gateway = 192.168.0.1 inspection_iprange = 192.168.0.100,192.168.0.120 masquerade = false #TODO(skatlapa): add param to override masq [stack@undercloud-0 ~]$ cat /etc/sysconfig/docker # /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs OPTIONS="--log-driver=journald --signature-verification=false" if [ -z "${DOCKER_CERT_PATH}" ]; then DOCKER_CERT_PATH=/etc/docker fi # Do not add registries in this file anymore. Use /etc/containers/registries.conf # instead. For more information reference the registries.conf(5) man page. # Location used for temporary files, such as those created by # docker load and build operations. Default is /var/lib/docker/tmp # Can be overriden by setting the following environment variable. # DOCKER_TMPDIR=/var/tmp # Controls the /etc/cron.daily/docker-logrotate cron job status. # To disable, uncomment the line below. # LOGROTATE=false # docker-latest daemon can be used by starting the docker-latest unitfile. # To use docker-latest client, uncomment below lines #DOCKERBINARY=/usr/bin/docker-latest #DOCKERDBINARY=/usr/bin/dockerd-latest #DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest #DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest INSECURE_REGISTRY="--insecure-registry 192.168.0.1:8787 --insecure-registry 192.168.0.3:8787"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0068