Bug 1641704 - Unable to upload container images, docker push fails
Summary: Unable to upload container images, docker push fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z4
: 13.0 (Queens)
Assignee: Emilien Macchi
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-10-22 14:50 UTC by Raoul Scarazzini
Modified: 2019-01-16 17:55 UTC (History)
8 users (show)

Fixed In Version: instack-undercloud-8.4.6-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-16 17:55:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1799784 0 None None None 2018-10-24 20:15:16 UTC
OpenStack gerrit 613406 0 None MERGED [queens-only] include local_ip & undercloud_admin_host in docker_insecure_registries 2021-01-24 18:56:04 UTC
Red Hat Product Errata RHBA-2019:0068 0 None None None 2019-01-16 17:55:20 UTC

Description Raoul Scarazzini 2018-10-22 14:50:01 UTC
Description of problem:

OSP13 is failing during containers preparation while trying to push containers on the local registry:

clean_up UploadImage: Could not push image 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 64, in take_action
    uploader.upload()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 96, in upload
    uploader.run_tasks()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 388, in run_tasks
    result = self.upload_image(*first)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 184, in upload_image
    DockerImageUploader._push(dockerc, new_repo, tag=tag)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 214, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
    start_time=start_time)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 265, in iter
    raise RetryError(fut).reraise()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 344, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 217, in _push
    raise ImageUploaderException('Could not push image %s' % image)
ImageUploaderException: Could not push image 192.168.24.1:8787/rhosp13/openstack-swift-proxy-server

The command used to complete the action is:

openstack overcloud container image upload --debug --config-file /home/stack/overcloud_containers.yaml

this command was used to generate overcloud_containers.yaml file:

openstack overcloud container image prepare \
    --output-images-file /home/stack/overcloud_containers.yaml \
    --output-env-file /home/stack/containers-default-parameters.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
    --namespace docker-registry.engineering.redhat.com/rhosp13 \
    --tag $BUILD_ID \
    --prefix "openstack" \
    --suffix "" \
    --set "ceph_namespace=docker.io/ceph" \
    --set "ceph_image=daemon" \
    --set "ceph_tag=v3.1.0-stable-3.1-luminous-centos-7-x86_64" \
    --push-destination 192.168.24.1:8787

The docker-registry.engineering.redhat.com is configured as INSECURE_REGISTRY inside /etc/sysconfig/docker.

Version-Release number of selected component (if applicable):

Puddle 2018-10-02.1

How reproducible:

The failure is consistent.

Steps to Reproduce:
1. Just generate the overcloud_containers.yaml file and launch the above openstack overcloud container image upload command

Actual results:

Could not push any image.

Expected results:

All the images pushed correctly.

Additional info:

sosreports on their way.

Comment 2 Steve Baker 2018-10-22 20:00:45 UTC
Can you please provide the following:
- generated overcloud_containers.yaml
- content of /etc/docker-distribution/registry/config.yml
- output of: sudo systemctl status docker-distribution

Comment 3 Raoul Scarazzini 2018-10-23 07:24:37 UTC
Hi Steve, here's what you requested:

[stack@undercloud ~]$ cat overcloud_containers.yaml
container_images:
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-evaluator:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-listener:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-notifier:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-central:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-compute:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-notification:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-scheduler:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-volume:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cron:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-glance-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-metricd:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-statsd:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-haproxy:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-api-cfn:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-engine:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-horizon:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-iscsid:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-keystone:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-mariadb:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-memcached:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-dhcp-agent:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-l3-agent:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-metadata-agent:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-openvswitch-agent:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-server:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-compute:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-conductor:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-consoleauth:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-libvirt:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-novncproxy:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-placement-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-scheduler:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-panko-api:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-rabbitmq:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-redis:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-account:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-container:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-object:2018-10-02.1
  push_destination: 192.168.24.1:8787
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-proxy-server:2018-10-02.1
  push_destination: 192.168.24.1:878

[stack@undercloud ~]$ cat /etc/docker-distribution/registry/config.yml
version: 0.1
log:
  fields:
    service: registry
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: 192.168.24.1:8787

[stack@undercloud ~]$ sudo systemctl status docker-distribution
● docker-distribution.service - v2 Registry server for Docker
   Loaded: loaded (/usr/lib/systemd/system/docker-distribution.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-10-22 07:23:18 EDT; 19h ago
 Main PID: 17684 (registry)
    Tasks: 13
   CGroup: /system.slice/docker-distribution.service
           └─17684 /usr/bin/registry serve /etc/docker-distribution/registry/config.yml

Oct 23 03:22:10 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:10 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:12 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:12 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:14 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:14 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:16 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:16 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:18 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:18 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:20 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:20 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:22 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:22 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:24 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:24 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:26 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:26 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""
Oct 23 03:22:28 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:28 -0400] "OPTIONS / HTTP/1.0" 200 0 "" ""

Comment 4 Steve Baker 2018-10-24 20:04:13 UTC
192.168.24.1:8787 should also be an insecure registry in /etc/sysconfig/docker.

To do this the undercloud.conf needs the following before the undercloud is installed:
docker_insecure_registries = 192.168.24.1:8787, docker-registry.engineering.redhat.com

To unblock the production chain, this is what I suggest, but we need to decide what to do for the customer.

OSP-14 adds 192.168.24.1:8787 automatically, but instack based OSP-13 doesn't. We can do one of the following:
- Document that they must always manually add docker_insecure_registries = 192.168.24.1:8787 to undercloud.conf
- Propose a fix to instack-undercloud to append local_ip:8787 DOCKER_INSECURE_REGISTRIES if it isn't there

Comment 9 Raoul Scarazzini 2018-11-19 13:08:50 UTC
Hey all, I'm still hitting this on the latest Puddle (2018-11-07.3) with package "instack-undercloud-8.4.6-2.el7ost.noarch", so not yet the one mentioned in "Fixed in version". What else needs to be done to make this move?

Comment 22 Marius Cornea 2018-12-10 20:09:10 UTC
[stack@undercloud-0 ~]$ cat undercloud.conf 
[DEFAULT]
# Network interface on the Undercloud that will be handling the PXE
# boots and DHCP for Overcloud instances. (string value)
local_interface = eth0
local_ip = 192.168.0.1/24
undercloud_public_host = 192.168.0.2
undercloud_admin_host = 192.168.0.3
#TODO: use release >= 10 when RHBZ#1633193 is resolved
undercloud_ntp_servers=clock.redhat.com
hieradata_override = /home/stack/hiera_override.yaml
[ctlplane-subnet]
local_subnet = ctlplane-subnet
cidr = 192.168.0.0/24
dhcp_start = 192.168.0.5
dhcp_end = 192.168.0.24
gateway = 192.168.0.1
inspection_iprange = 192.168.0.100,192.168.0.120
masquerade = false
#TODO(skatlapa): add param to override masq
[stack@undercloud-0 ~]$ cat /etc/sysconfig/docker
# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS="--log-driver=journald --signature-verification=false"
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi

# Do not add registries in this file anymore. Use /etc/containers/registries.conf
# instead. For more information reference the registries.conf(5) man page.

# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp

# Controls the /etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# LOGROTATE=false

# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKERDBINARY=/usr/bin/dockerd-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest
INSECURE_REGISTRY="--insecure-registry 192.168.0.1:8787 --insecure-registry 192.168.0.3:8787"

Comment 25 errata-xmlrpc 2019-01-16 17:55:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0068


Note You need to log in before you can comment on or make changes to this bug.