Bug 1641704
Summary: | Unable to upload container images, docker push fails | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Raoul Scarazzini <rscarazz> |
Component: | instack-undercloud | Assignee: | Emilien Macchi <emacchi> |
Status: | CLOSED ERRATA | QA Contact: | Arik Chernetsky <achernet> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 13.0 (Queens) | CC: | bdobreli, emacchi, jschluet, m.andre, mariel, mburns, mcornea, sbaker |
Target Milestone: | z4 | Keywords: | Triaged, ZStream |
Target Release: | 13.0 (Queens) | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | instack-undercloud-8.4.6-3.el7ost | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-01-16 17:55:05 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Raoul Scarazzini
2018-10-22 14:50:01 UTC
Can you please provide the following: - generated overcloud_containers.yaml - content of /etc/docker-distribution/registry/config.yml - output of: sudo systemctl status docker-distribution Hi Steve, here's what you requested: [stack@undercloud ~]$ cat overcloud_containers.yaml container_images: - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-evaluator:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-listener:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-aodh-notifier:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-central:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-compute:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-ceilometer-notification:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-scheduler:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cinder-volume:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-cron:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-glance-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-metricd:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-gnocchi-statsd:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-haproxy:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-api-cfn:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-heat-engine:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-horizon:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-iscsid:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-keystone:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-mariadb:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-memcached:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-dhcp-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-l3-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-metadata-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-openvswitch-agent:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-neutron-server:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-compute:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-conductor:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-consoleauth:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-libvirt:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-novncproxy:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-placement-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-nova-scheduler:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-panko-api:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-rabbitmq:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-redis:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-account:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-container:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-object:2018-10-02.1 push_destination: 192.168.24.1:8787 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-swift-proxy-server:2018-10-02.1 push_destination: 192.168.24.1:878 [stack@undercloud ~]$ cat /etc/docker-distribution/registry/config.yml version: 0.1 log: fields: service: registry storage: cache: layerinfo: inmemory filesystem: rootdirectory: /var/lib/registry http: addr: 192.168.24.1:8787 [stack@undercloud ~]$ sudo systemctl status docker-distribution ● docker-distribution.service - v2 Registry server for Docker Loaded: loaded (/usr/lib/systemd/system/docker-distribution.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2018-10-22 07:23:18 EDT; 19h ago Main PID: 17684 (registry) Tasks: 13 CGroup: /system.slice/docker-distribution.service └─17684 /usr/bin/registry serve /etc/docker-distribution/registry/config.yml Oct 23 03:22:10 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:10 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:12 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:12 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:14 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:14 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:16 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:16 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:18 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:18 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:20 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:20 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:22 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:22 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:24 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:24 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:26 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:26 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" Oct 23 03:22:28 undercloud registry[17684]: 192.168.24.1 - - [23/Oct/2018:03:22:28 -0400] "OPTIONS / HTTP/1.0" 200 0 "" "" 192.168.24.1:8787 should also be an insecure registry in /etc/sysconfig/docker. To do this the undercloud.conf needs the following before the undercloud is installed: docker_insecure_registries = 192.168.24.1:8787, docker-registry.engineering.redhat.com To unblock the production chain, this is what I suggest, but we need to decide what to do for the customer. OSP-14 adds 192.168.24.1:8787 automatically, but instack based OSP-13 doesn't. We can do one of the following: - Document that they must always manually add docker_insecure_registries = 192.168.24.1:8787 to undercloud.conf - Propose a fix to instack-undercloud to append local_ip:8787 DOCKER_INSECURE_REGISTRIES if it isn't there Hey all, I'm still hitting this on the latest Puddle (2018-11-07.3) with package "instack-undercloud-8.4.6-2.el7ost.noarch", so not yet the one mentioned in "Fixed in version". What else needs to be done to make this move? [stack@undercloud-0 ~]$ cat undercloud.conf [DEFAULT] # Network interface on the Undercloud that will be handling the PXE # boots and DHCP for Overcloud instances. (string value) local_interface = eth0 local_ip = 192.168.0.1/24 undercloud_public_host = 192.168.0.2 undercloud_admin_host = 192.168.0.3 #TODO: use release >= 10 when RHBZ#1633193 is resolved undercloud_ntp_servers=clock.redhat.com hieradata_override = /home/stack/hiera_override.yaml [ctlplane-subnet] local_subnet = ctlplane-subnet cidr = 192.168.0.0/24 dhcp_start = 192.168.0.5 dhcp_end = 192.168.0.24 gateway = 192.168.0.1 inspection_iprange = 192.168.0.100,192.168.0.120 masquerade = false #TODO(skatlapa): add param to override masq [stack@undercloud-0 ~]$ cat /etc/sysconfig/docker # /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs OPTIONS="--log-driver=journald --signature-verification=false" if [ -z "${DOCKER_CERT_PATH}" ]; then DOCKER_CERT_PATH=/etc/docker fi # Do not add registries in this file anymore. Use /etc/containers/registries.conf # instead. For more information reference the registries.conf(5) man page. # Location used for temporary files, such as those created by # docker load and build operations. Default is /var/lib/docker/tmp # Can be overriden by setting the following environment variable. # DOCKER_TMPDIR=/var/tmp # Controls the /etc/cron.daily/docker-logrotate cron job status. # To disable, uncomment the line below. # LOGROTATE=false # docker-latest daemon can be used by starting the docker-latest unitfile. # To use docker-latest client, uncomment below lines #DOCKERBINARY=/usr/bin/docker-latest #DOCKERDBINARY=/usr/bin/dockerd-latest #DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest #DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest INSECURE_REGISTRY="--insecure-registry 192.168.0.1:8787 --insecure-registry 192.168.0.3:8787" Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0068 |