Bug 1663451

Summary: oc new-app -S in <nonexistentimage> a disconnected environment searches in registry-1.docker.io
Product: OpenShift Container Platform Reporter: Joel Rosental R. <jrosenta>
Component: BuildAssignee: Ben Parees <bparees>
Status: CLOSED WONTFIX QA Contact: wewang <wewang>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, wzheng
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1974754 (view as bug list) Environment:
Last Closed: 2019-02-21 22:39:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1974754    

Description Joel Rosental R. 2019-01-04 11:35:46 UTC
Description of problem:
Using `oc new-app -S <is>` and <id> refers to an non-existent imagestream or image in a disconnected environent it always return an error like this:

"$ oc new-app -S testimagethatdoesntexist
W0104 06:19:27.846414   36235 dockerimagelookup.go:233] Docker registry lookup failed: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
error: no matches found"

It looks like it always try to look in registry-1.docker.io even if that registry is not listed on any docker config file, e.g: /etc/containers/registries.conf or /etc/sysconfig/docker:

# docker pull registry-1.docker.io/testimagethatdoesntexist
Using default tag: latest
Trying to pull repository registry-1.docker.io/testimagethatdoesntexist ... 
All endpoints blocked.

# cat /etc/containers/registries.conf
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: 'registries.search', 'registries.insecure', 
# and 'registries.block'.

[registries.search]
registries = []

# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = []


# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = []


# cat /etc/sysconfig/docker
# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS=' --selinux-enabled       --log-driver json-file --log-opt max-size=2M --log-opt max-file=3 --ipv6=false --signature-verification=False'
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi

# Do not add registries in this file anymore. Use /etc/containers/registries.conf
# instead. For more information reference the registries.conf(5) man page.

# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp

# Controls the /etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# LOGROTATE=false

# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKERDBINARY=/usr/bin/dockerd-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest
ADD_REGISTRY='--add-registry registry.kvm.local --add-registry registry.redhat.io'
BLOCK_REGISTRY='--block-registry all'
INSECURE_REGISTRY='--insecure-registry 10.36.0.0/16'

root      43352      1  2 00:11 ?        00:00:06 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver json-file --log-opt max-size=2M --log-opt max-file=3 --ipv6=false --signature-verification=False --storage-driver overlay2 --mtu=1450 --add-registry registry.kvm.local --add-registry registry.redhat.io --block-registry all --insecure-registry 10.36.0.0/16

Version-Release number of selected component (if applicable):

$ rpm -q docker
docker-1.13.1-88.git07f3374.el7.x86_64

Tested in:

OCP v3.10.83 and v3.11.51

How reproducible:
Always

Steps to Reproduce:
In a disconnected/offline environment:

1. In a disconnected environment: `oc new-app -S testimagethatdoesntexist`

In a non-disconnected environment:
1. Either block registry-1.docker.io in /etc/containers/registries.conf or add a bogus ip for registry-1.docker.io in /etc/hosts
2. Restart dnsmasq: `systemctl restart dnsmasq.service`
3. `oc new-app -S testimagethatdoesntexist`

Actual results:
W0104 06:19:27.846414   36235 dockerimagelookup.go:233] Docker registry lookup failed: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
error: no matches found

Expected results:
It should not look in registry-1.docker.io

Additional info:
Looks like a regression bug for: https://bugzilla.redhat.com/show_bug.cgi?id=1398330

Comment 1 Ben Parees 2019-02-21 22:39:32 UTC
> It looks like it always try to look in registry-1.docker.io even if that registry is not listed on any docker config file, e.g: /etc/containers/registries.conf or /etc/sysconfig/docker:

The decision about where to try to look for the image is part of the openshift server itself, it's not configurable (docker.io is always the default for image names which do not specify a registry):

https://github.com/openshift/origin/blob/d45151824821195c9d3db7e5d2da0a1c982a614c/pkg/image/api/helper.go#L23-L36

And it's not controllable from the client side because we're making a request to the server to lookup the image for us, we have no way of knowing whether the server can reach docker.io or not (and in fact the server itself may not have a way of knowing).

The previous work on this bug lead to the current state you see today (a warning message being reported):

https://bugzilla.redhat.com/show_bug.cgi?id=1398330
https://bugzilla.redhat.com/show_bug.cgi?id=1378647

I don't think there is any further worth doing here (not even sure what's reasonably possible).