Description of problem: I did a fresh installation of version 4.4.0.3. After the engine setup I replaced the apache certificate with a custom certificate. I used this article to do it: https://myhomelab.gr/linux/2020/01/20/replacing_ovirt_ssl.html To summarize, I replaced those files with my own authority and the signed custom certificate /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/apache-ca.pem That worked so far, apache uses now my certificate, login is possible. To setup a new machine, I need to upload an iso image, which failed. I found this error in /var/log/ovirt-imageio/daemon.log 2020-07-08 20:43:23,750 INFO (Thread-10) [http] OPEN client=192.168.1.228 2020-07-08 20:43:23,767 INFO (Thread-10) [backends.http] Open backend netloc='the_secret_hostname:54322' path='/images/ef60404c-dc69-4a3d-bfaa-8571f675f3e1' cafile='/etc/pki/ovirt-engine/apache-ca.pem' secure=True 2020-07-08 20:43:23,770 ERROR (Thread-10) [http] Server error Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 699, in __call__ self.dispatch(req, resp) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 744, in dispatch return method(req, resp, *match.groups()) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/cors.py", line 84, in wrapper return func(self, req, resp, *args) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/images.py", line 66, in put backends.get(req, ticket, self.config), File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py", line 53, in get cafile=config.tls.ca_file) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 48, in open secure=options.get("secure", True)) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 63, in __init__ options = self._options() File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/http.py", line 364, in _options self._con.request("OPTIONS", self.url.path) File "/usr/lib64/python3.6/http/client.py", line 1254, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 974, in send self.connect() File "/usr/lib64/python3.6/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 776, in __init__ self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) 2020-07-08 20:43:23,770 INFO (Thread-10) [http] CLOSE client=192.168.1.228 [connection 1 ops, 0.019775 s] [dispatch 1 ops, 0.003114 s] I'm a python developer so I had no problem reading the traceback. The SSL handshake fails when image-io tries to connect to what I think is called an ovn-provider. But it is using my new authority certificate cafile='/etc/pki/ovirt-engine/apache-ca.pem' which does not validate the certificate generated by the ovirt engine setup, which the ovn-provider probably uses. I didn't exactly know where the parameter for the validation ca file is. Probably it is the ca_file parameter in /etc/ovirt-imageio/conf.d/50-engine.conf. But that needs to be set to my own authority ca file. I modified the python file to set the ca_file parameter to the engine setups ca_file directly /usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/backends/__init__.py So the function call around line 50 looks like this: backend = module.open( ticket.url, mode, sparse=ticket.sparse, dirty=ticket.dirty, cafile='/etc/pki/ovirt-engine/ca.pem' #config.tls.ca_file ) Now the image upload works, but obviously this is not the way to fix things. Is there an other way to make image-io accept the certificate from the engine setup, while using my custom certificate? I don't want to replace the certificates of all ovirt components with custom certificates. I only need the weblogin with my custom certificate. Bug report based on this thread in ovirt users list: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/365ISJ7JHAKYIGYPQFXUGDBS7UHJDLI7/ Version-Release number of selected component (if applicable): 2.0.9 How reproducible: Awlays Steps to Reproduce: 1. Replace apache certificates generated by engine by other certificates signed by another CA 2. Restart ovit-imageio service 3. Try to upload or download images Actual results: Transfer fail when accessing imageio daemon on the host because imageio daemon on engine side use the apache CA to verify host imageio daemon certificate. Expected results: imageio daemon on engine side should use engine CA to verify host server certificate.
In ovirt-4.3 the proxy had both appache ca_file an engine ca_file, and the engine cert_file was used when accessing the host: 226 imaged_prepped = imaged_session.prepare_request(imaged_req) 227 imaged_resp = imaged_session.send( 228 imaged_prepped, verify=config.engine_ca_cert_file, 229 timeout=(connection_timeout, read_timeout), stream=stream) Because the certificate was self signed and it included engine ca, it could be used to verify the host certificates, which are vdsm certificates. In 4.4 we have only one ca_file. By default is the apache ca_file generated by engine, so it works for verifying host certificates, but if you replace it with another CA certificate, it will fail to verity the host certificates. I think the change needed is to add ca_file for the http backend, used to communicate with hosts on the proxy side. This must always be the engine CA certificate since we don't support user certificates on the hosts. Suggested changes: imageio: - Add pki files for proxy, currently we have only one set of pki files - Use proxy pki files in the proxy tests - this will reproduce the issue - Add configuration for http backend: [backend_http] ca_file = - Use config.backend_http.ca_file for http backend, instead of config.tls.ca_file - In the proxy tests, configure http backend ca_file to daemon ca_file. - Document new settings in docs/configuration.md engine: - Include backend_http:ca_file in imageio configuration (/etc/ovirt-imageio/conf.d/50-engine.conf) Handling upgrade: - When host is upgraded, engine setup will create new 50-engine.conf enabling the new configuration. This should keep user configured tls settings in 99-user.conf. Didi, what do you think?
Actually this is not about the administration portal, image transfer using the API/SDK via the proxy is also broken after replacing builtin certificates with 3rd party certificates.
> Handling upgrade: > - When host is upgraded, engine setup will create new 50-engine.conf > enabling the new configuration. This should keep user configured > tls settings in 99-user.conf. in case user set up own certificates, 99-user.conf needs to be modified with [backend_http] ca_file = /path/to/user/cert otherwise dafault certificates will be used. Probably we should use same value as for tls.ca_file if no value is set. In case user wants to use default certificates, we will use default here as well.
Setting backend_http:ca_file is not needed in oVirt setup since we don't support user certificates on the hosts.
(In reply to Nir Soffer from comment #4) > Setting backend_http:ca_file is not needed in oVirt setup since > we don't support user certificates on the hosts. I'd keep the defaulting to tls.ca_file anyway as it make config more easy and backward compatible (adding backend_http:ca_file is not needed)
Do you mean that if backend_http.ca_file is not configured, use tls.ca_file? This will break when you replace certificates, so this is not a good idea. Maybe if you long a very clear warning about missing configuration. Regardless, engine *must* add backend_http:ca_file to 50-engine.conf. Vdsm does not need to configure this since we never add proxy tickets on a host.
Looks good to me. Not sure the naming is very clear: - backend_http.ca_file is for accessing the engine, right? - tls.ca_file is for accessing hosts, right? And both point at CA Cert files. I'd personally expect the new setting to be under exiting section tls, perhaps named "backend_ca_file" if you want something similar to your proposal. I'd not use 'http' when you actually mean 'https'. 'http', today, sounds as deliberately "not https". To me. Re upgrades and using 3rd-party certs for https: We should (and already do, AFAIU) use, for tls.ca_file, a (by default) symlink to engine's ca cert, called apache-ca.pem. In the docs to use 3rd-party certs, we say to replace this symlink with the cert of the ca we want to use. So this should not require editing conf files, and work across upgrades.
(In reply to Yedidyah Bar David from comment #7) > Looks good to me. > > Not sure the naming is very clear: > - backend_http.ca_file is for accessing the engine, right? > - tls.ca_file is for accessing hosts, right? The other way: - tls.ca_file: for the ovirt-imageio service. This is is by default apache ca_file, and the user can replace the file or change the configuration as needed to use whatever file they want. This must match the cafile used by the client talking with the proxy. - backend_http.ca_file: for contacting the ovirt-imageio service on the host. This is by default engine ca_file, and should never change, since we don't support user pki on the hosts. Users should never change or care about backend_http.ca_file, only about tls.* options. > And both point at CA Cert files. > > I'd personally expect the new setting to be under exiting section tls, > perhaps named "backend_ca_file" if you want something similar to your > proposal. tls is the public configuration that users should be aware of. backned_http.cal_file is something they should not care about. We also need other conflagration for backendds, so we are going to add backend_file and backend_nbd sections soon. Keeping all things related to specific backend at the same place is better than adding different kinds of configurations options under tls section. > I'd not use 'http' when you actually mean 'https'. 'http', today, > sounds as deliberately "not https". To me. https is http + tls, the difference is not important. Actually we support only https, but we use a more generic name for the backend. In the same way we have file backend that handles file and block devices and nbd backend handling nbd:// (tcp) and nbd+unix:// URLs. The backend specific configuration is something that users should not be aware of care about. This is more useufl for development and testing, being able to tune the system without changing the code. > Re upgrades and using 3rd-party certs for https: > > We should (and already do, AFAIU) use, for tls.ca_file, a (by default) > symlink to engine's ca cert, called apache-ca.pem. This is good default. > In the docs to use 3rd-party certs, we say to replace this symlink > with the cert of the ca we want to use. So this should not require > editing conf files, and work across upgrades. This should work, but I think a better way is to leave the files engine created and point to your own files elsewhere using a drop-in configuration. This way engine will never overwrite your certs by accident since it does not know abut them.
I forgot to add that the http backend supports also http over unix socket. This is by imageio client to optimize local transfer when running on a hypervisor. So for something supporting https, http+unix http is a good name.
Verified in ovirt-imageio-daemon-2.0.10-1.el8ev.x86_64 Download and upload of an image is working with 3rd party CA and changed Apache certificates
This bugzilla is included in oVirt 4.4.2 release, published on September 17th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.2 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.