Bug 1689230

Summary: openshift_docker_blocked_registries is not blocking registries
Product: OpenShift Container Platform Reporter: Shivkumar Ople <sople>
Component: InstallerAssignee: Joseph Callen <jcallen>
Installer sub component: openshift-ansible QA Contact: Weihua Meng <wmeng>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: gpei, wmeng
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.11.102-1.git.0.16a8aac.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-26 09:07:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Shivkumar Ople 2019-03-15 13:28:56 UTC
Description of problem:

When mentioned the variable "openshift_docker_blocked_registries" in the inventory with 'all' value assigned(while doing fresh installation), it's not blocking the registries at all.
As per the usage of this variable, it should block the registries except for the allowed registries. But still, the registries.block section in the /etc/containers/registries.conf is empty.  


Installation steps:

1) Followed Host preparation guide: https://docs.openshift.com/container-platform/3.11/install/host_preparation.html
2) Ran the prerequisites.yaml playbook from /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml
3) Ran the deploy_cluster.yml from /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml





Version-Release number of the following components:

rpm -q openshift-ansible - openshift-ansible-3.11.69-1.git.0.2ff281f.el7.noarch

rpm -q ansible - ansible-2.6.12-1.el7ae.noarch

ansible --version - ansible 2.6.12
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/ocp-mgmt/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

Always

Steps to Reproduce:
1. Mention openshift_docker_blocked_registries parameter in the registry with value "all"

2. Run the prerequisites.yml, deploy_cluster.yaml

3. After installation check, if registry.redhat.io or other registries are accessible or not which are not included in the openshift_docker_additional_registries

Actual results:

Registries are accessible after blocking.


Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2019-03-15 13:55:37 UTC
Please, whenever possible, include actual inputs and outputs rather than describing them which is open to misinterpretation.

Reviewing the code this will populate the values into /etc/sysconfig/docker by default rather than /etc/containers/registries.conf. Can you confirm that's the case and that the settings defined therein are taking effect?
The output of `docker info` will illustrate how exactly docker is configured.

Comment 2 Shivkumar Ople 2019-03-18 12:27:56 UTC
Hello,


(In reply to Scott Dodson from comment #1)
> Please, whenever possible, include actual inputs and outputs rather than
> describing them which is open to misinterpretation.
> 
> Reviewing the code this will populate the values into /etc/sysconfig/docker
> by default rather than /etc/containers/registries.conf. Can you confirm
> that's the case and that the settings defined therein are taking effect?
> The output of `docker info` will illustrate how exactly docker is configured.

Yes, we can confirm that's the case.

--add-registry registry.redhat.io <--- This value should not be there in ADD_REGISTRY, after assigning "all" to "openshift_docker_blocked_registries" this variable.

In this case, it's assigned in the below manner.

~~~~
ADD_REGISTRY='--add-registry <internal-artifactory> --add-registry docker-registry.default.svc:5000 --add-registry registry.redhat.io'
BLOCK_REGISTRY='--block-registry all'
INSECURE_REGISTRY='--insecure-registry <internal-artifactory>'
~~~~

Thank you

Comment 3 Scott Dodson 2019-03-18 12:33:46 UTC
Ok, we append that for all installs where openshift_deployment_type == 'openshift-enterprise'. I guess it makes sense to only do this when (openshift_deployment_type == 'openshift-enterprise' && oreg_url matches 'registry.redhat.io').

Can you please set an appropriate severity/priority based on your customer's opinion of this matter? It will then be worked on based on PM Score ordering.

Comment 4 Joseph Callen 2019-03-22 17:30:08 UTC
Submitted PR: https://github.com/openshift/openshift-ansible/pull/11390

Comment 9 Weihua Meng 2019-04-25 10:19:23 UTC
1. The PR is not good. https://github.com/openshift/openshift-ansible/pull/11390
oreg_url is optional, 
for installation, 
if oreg_url is not set and openshift_docker_blocked_registries=all
install will fail as the registry is blocked to pull images


2. I think the logic would be something like:
when openshift_docker_blocked_registries=all,
all registries other than (listed in openshift_docker_additional_registries  plus registry in oreg_url) are blocked.
if oreg_url not set and openshift_deployment_type=openshift-enterprise, default registry(registry.redhat.io) is allowed, unless explicitily listed in openshift_docker_blocked_registries.

Comment 14 Joseph Callen 2019-05-03 13:48:32 UTC
PR:  https://github.com/openshift/openshift-ansible/pull/11565
Build: openshift-ansible-3.11.112-1

Comment 17 Weihua Meng 2019-06-17 09:24:28 UTC
Failed.

openshift-ansible-3.11.119-1.git.0.c9a8ebf.el7


oreg_url=registry.reg-aws.openshift.com:443/openshift3/ose-${component}:${version}
openshift_docker_insecure_registries=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888,virt-openshift-05.lab.eng.nay.redhat.com:5000,virt-openshift-05.lab.eng.nay.redhat.com:5001,registry.reg-aws.openshift.com:443,asb-registry.usersys.redhat.com:5000
openshift_docker_blocked_registries=all


        "Jun 17 05:13:57 ip-172-18-8-21.ec2.internal atomic-openshift-node[13414]: E0617 05:13:57.028197   13414 kuberuntime_manager.go:646] createPodSandbox for pod \"master-api-ip-172-18-8-21.ec2.internal_kube-system(9d066f84b20195c767ec4ed9d7ac3ba2)\" failed: rpc error: code = Unknown desc = All endpoints blocked.", 



[root@ip-172-18-8-21 ~]# cat /etc/containers/registries.conf 
# Ansible managed
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.

# The default location for this configuration file is /etc/containers/registries.conf.

# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.

[registries.search]
registries = []


# If you need to access insecure registries, add the registry's fully-qualified name.
# An insecure registry is one that does not have a valid SSL certificate or only does HTTP.
[registries.insecure]
registries = ["brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888", "virt-openshift-05.lab.eng.nay.redhat.com:5000", "virt-openshift-05.lab.eng.nay.redhat.com:5001", "registry.reg-aws.openshift.com:443", "asb-registry.usersys.redhat.com:5000"]


# If you need to block pull access from a registry, uncomment the section below
# and add the registries fully-qualified name.
#
# Docker only
[registries.block]
registries = ["all"]
[root@ip-172-18-8-21 ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
[root@ip-172-18-8-21 ~]# docker pull asd
Using default tag: latest
Error response from daemon: No configured registry to pull from.
[root@ip-172-18-8-21 ~]# docker pull registry.reg-aws.openshift.com:443/openshift3/ose-node:v3.11
Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-node ... 
All endpoints blocked.

Comment 19 Weihua Meng 2019-06-18 01:44:48 UTC
Fixed.

openshift-ansible-3.11.119-1.git.0.c9a8ebf.el7

Be aware that when openshift_docker_blocked_registries=all set, 
if oreg_url is specified, must set the registry used in oreg_url to openshift_docker_additional_registries, or install will fail due to registry blocked.

Comment 21 errata-xmlrpc 2019-06-26 09:07:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1605