Description of problem: docker fails to start on these AMIs due to the following entry in /etc/containers/registries.conf >>>> # If you need to access insecure registries, add the registry's fully-qualified name. # An insecure registry is one that does not have a valid SSL certificate or only does HTTP. [registries.insecure] registries = [""] <<<< When this entry is present, /var/run/containers/registries.conf is computed by docker as follows: >>>> REGISTRIES="--add-registry registry.redhat.io --add-registry docker.io --insecure-registry " <<<< The failure to include anything after --insecure-registry makes docker fail to start. >>>> Nov 14 12:58:48 ip-172-16-0-24.ec2.internal systemd[1]: Starting Docker Application Container Engine... Nov 14 12:58:48 ip-172-16-0-24.ec2.internal dockerd-current[4419]: Status: flag needs an argument: --insecure-registry Nov 14 12:58:48 ip-172-16-0-24.ec2.internal dockerd-current[4419]: See 'dockerd --help'. Nov 14 12:58:48 ip-172-16-0-24.ec2.internal dockerd-current[4419]: Usage: dockerd COMMAND Nov 14 12:58:48 ip-172-16-0-24.ec2.internal dockerd-current[4419]: A self-sufficient runtime for containers. Nov 14 12:58:48 ip-172-16-0-24.ec2.internal dockerd-current[4419]: Options: .... <<<< Version-Release number of the following components: v3.11.44. This issue did not exist in v3.11.16. How reproducible: 100% Actual results: Docker fails to start. Expected results: If the /etc/containers/registries.conf entry is changed to the following, docker will not include the invalid --insecure-registry argument in the systemd invocation ("" is removed from [""]). >>>> [registries.insecure] registries = [] <<<< Additional info: This configuration results from building a CRI-O AMI using openshift-ansible (openshift-ansible/playbooks/aws/openshift-cluster/build_ami.yml), but it is possible other flows are affected. Docker is required on CRI-O nodes for OpenShift builds.
We have created an AMI with the fix contained in the following PR: https://github.com/openshift/openshift-ansible/pull/10799 We are seeing the following content in /etc/containers/registries.conf: [registries.search] registries = [["registry.redhat.io", "docker.io"]] # If you need to access insecure registries, add the registry's fully-qualified name. # An insecure registry is one that does not have a valid SSL certificate or only does HTTP. [registries.insecure] registries = [[]] The extra set of embedded brackets are preventing docker from starting properly.
In openshift-ansible-3.11.58-1 and later.
Could reproduce this bug with openshift-ansible-3.11.44-1.git.0.11d174e.el7.noarch.rpm When setting up a cri-o&docker mixed env, no openshift_docker_insecure_registries specified in ansible inventory file, installer will leave '[""]' value in [registries.insecure] of /etc/containers/registries.conf [root@ip-172-18-10-241 ~]# grep '\[registries.insecure\]' -A 1 /etc/containers/registries.conf [registries.insecure] registries = [""] Then docker service failed to start for invalid argument in /var/run/containers/registries.conf Jan 04 02:03:15 ip-172-18-10-241.ec2.internal systemd[1]: Starting Docker Application Container Engine... Jan 04 02:03:15 ip-172-18-10-241.ec2.internal dockerd-current[22185]: Status: flag needs an argument: --insecure-registry [root@ip-172-18-10-241 ~]# cat /var/run/containers/registries.conf REGISTRIES="--add-registry registry.redhat.io --add-registry docker.io --insecure-registry " Verified with openshift-ansible-3.11.65-1.git.0.6a0837b.el7.noarch.rpm. With PR https://github.com/openshift/openshift-ansible/pull/10799 applied, the default value of [registries.insecure] turns into []. [root@ip-172-18-5-125 ~]# grep '\[registries.insecure\]' -A 1 /etc/containers/registries.conf [registries.insecure] registries = [] [root@ip-172-18-5-125 ~]# cat /var/run/containers/registries.conf REGISTRIES="--add-registry registry.access.redhat.com --add-registry docker.io --add-registry registry.fedoraproject.org --add-registry quay.io --add-registry registry.centos.org " cri-o and docker service both run well.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0096