Description of problem: Running a local container registry with basic authentication and TLS encryption causes problems with the "openstack tripleo container image prepare" command. The authentication and encryption is needed because the same registry is used for OpenShift airgapped deployment. Version-Release number of selected component (if applicable): 16.1 How reproducible: ----> setting up the container registry Creating the container registry to mirror OpenStack Start by creating the space to run the container registry mkdir -p /opt/registry/{auth,certs} mkdir /var/www/html/repo/containers Create the TLS certs: cd /opt/registry/certs openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 3650 -out domain.crt --- output as example -> Generating a RSA private key .....................................................................................................++++ ......................................................................................................................................................................................................................................................++++ writing new private key to 'domain.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:NL State or Province Name (full name) []:ZH Locality Name (eg, city) [Default City]:Den Haag Organization Name (eg, company) [Default Company Ltd]:Red Hat Organizational Unit Name (eg, section) []:Container Service Common Name (eg, your name or your server's hostname) []:repo-server Email Address []:bart <- end of example output --- Create the user to access the registry sudo chown -R $USER /opt/registry htpasswd -bBc /opt/registry/auth/htpasswd openshift redhat Start the registry podman run -d --name mirror-registry \ -p 5000:5000 --restart=always \ -v /var/www/html/repo/containers:/var/lib/registry:z \ -v /opt/registry/auth:/auth:z \ -e "REGISTRY_AUTH=htpasswd" \ -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \ -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \ -v /opt/registry/certs:/certs:z \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \ -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \ docker.io/library/registry:2 Make sure the CA is added to the trust chain cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust ----> setting up the mirror procedure sudo yum install -y python3-tripleoclient Generate a containers-prepare-parameter.yaml openstack tripleo container image prepare default --local-push-destination --output-env-file containers-prepare-parameter.yaml Add the credentials and push_destination: parameter_defaults: ContainerImagePrepare: - push_destination: repo-server:5000 set: ceph_alertmanager_image: ose-prometheus-alertmanager ceph_alertmanager_namespace: registry.redhat.io/openshift4 ceph_alertmanager_tag: 4.1 ceph_grafana_image: rhceph-4-dashboard-rhel8 ceph_grafana_namespace: registry.redhat.io/rhceph ceph_grafana_tag: 4 ceph_image: rhceph-4-rhel8 ceph_namespace: registry.redhat.io/rhceph ceph_node_exporter_image: ose-prometheus-node-exporter ceph_node_exporter_namespace: registry.redhat.io/openshift4 ceph_node_exporter_tag: v4.1 ceph_prometheus_image: ose-prometheus ceph_prometheus_namespace: registry.redhat.io/openshift4 ceph_prometheus_tag: 4.1 ceph_tag: latest name_prefix: openstack- name_suffix: '' namespace: registry.redhat.io/rhosp-rhel8 neutron_driver: ovn rhel_containers: false tag: '16.1' tag_from_label: '{version}-{release}' ContainerImageRegistryCredentials: registry.redhat.io: rhn_username: 'rhn_password' repo-server: openshift: 'redhat' Run the prepare command to initiate the mirror action: sudo openstack tripleo container image prepare --environment-file /home/stack/containers-prepare-parameter.yaml Actual results: sudo openstack tripleo container image prepare --environment-file /home/stack/containers-prepare-parameter.yaml /usr/lib/python3.6/site-packages/urllib3/connection.py:362: SubjectAltNameWarning: Certificate for repo-server has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning namedtuple_ParseResult:14: ResourceWarning: unclosed <ssl.SSLSocket fd=13, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('192.168.178.41', 60206), raddr=('192.168.178.42', 5000)> Exception occured while running the command concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib64/python3.6/concurrent/futures/process.py", line 175, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/lib64/python3.6/concurrent/futures/process.py", line 153, in _process_chunk return [fn(*args) for args in chunk] File "/usr/lib64/python3.6/concurrent/futures/process.py", line 153, in <listcomp> return [fn(*args) for args in chunk] File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2489, in upload_task return uploader.upload_image(task) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 1386, in upload_image password=target_password File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 292, in wrapped_f return self.call(f, *args, **kw) File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 358, in call do = self.iter(retry_state=retry_state) File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 319, in iter return fut.result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 361, in call result = fn(*args, **kwargs) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 610, in authenticate 'Unknown www-authenticate value: %s' % www_auth) tripleo_common.image.exception.ImageUploaderException: Unknown www-authenticate value: Basic realm="Registry Realm" """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/container_image.py", line 1043, in take_action cleanup=parsed_args.cleanup, lock=lock) File "/usr/lib/python3.6/site-packages/tripleo_common/image/kolla_builder.py", line 235, in container_images_prepare_multi uploader.upload() File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 443, in upload uploader.run_tasks() File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_uploader.py", line 2430, in run_tasks for result in p.map(upload_task, self.upload_tasks): File "/usr/lib64/python3.6/concurrent/futures/process.py", line 366, in _chain_from_iterable_of_lists for element in iterable: File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 586, in result_iterator yield fs.pop().result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception tripleo_common.image.exception.ImageUploaderException: Unknown www-authenticate value: Basic realm="Registry Realm" Unknown www-authenticate value: Basic realm="Registry Realm" Expected results: Fully synced container mirror Additional info: Could that i need different settings for the registry TLS certificate
So a few issues here is that if you have a custom ssl cert you will need to add the server to the DockerInsecureRegistryAddress. Additionally you will likely need to properly configure podman with the insecure registry. We'll have to look into it, but it may be that the registry:v2 doesn't support the same authentication process as the rest of the registries. Since this is not an officially supported thing at the moment since we do not currently test this process against anything but the undercloud, we'll look into if it is something that we can support.
Thank you Alex. I'll see if i can generate my CA and certs in a more acceptable manner.
So we currently do not support basic auth as a push destination. I will look into implementing this and backporting it. Once that functionality has been added, you would need to specify the credentials and add the repo server into the DockerInsecureRegistryAddress configuration. Note that for credentials you need to ensure the port is listed. Example: parameter_defaults: ContainerImagePrepare: - push_destination: repo-server:5000 set: ceph_alertmanager_image: ose-prometheus-alertmanager ceph_alertmanager_namespace: registry.redhat.io/openshift4 ceph_alertmanager_tag: 4.1 ceph_grafana_image: rhceph-4-dashboard-rhel8 ceph_grafana_namespace: registry.redhat.io/rhceph ceph_grafana_tag: 4 ceph_image: rhceph-4-rhel8 ceph_namespace: registry.redhat.io/rhceph ceph_node_exporter_image: ose-prometheus-node-exporter ceph_node_exporter_namespace: registry.redhat.io/openshift4 ceph_node_exporter_tag: v4.1 ceph_prometheus_image: ose-prometheus ceph_prometheus_namespace: registry.redhat.io/openshift4 ceph_prometheus_tag: 4.1 ceph_tag: latest name_prefix: openstack- name_suffix: '' namespace: registry.redhat.io/rhosp-rhel8 neutron_driver: ovn rhel_containers: false tag: '16.1' tag_from_label: '{version}-{release}' ContainerImageRegistryCredentials: "repo-server:5000": openshift: 'redhat' DockerInsecureRegistryAddress: - repo-server:5000
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:3483