Testing upload shows that reading from SSL socket is our bottleneck with fast storage. Using http over unix socket, we can eliminate this overhead when the upload is performed locally. Required changes: * Engine - expose a unix socket attribute in ImageTransfer entity (when the host supports this feature). * vdsm - return imageio info on getCaps. * imageio-daemon - service ‘/info’ over https and unix control socket.
severity?
We will start by exposing the unix socket only from OPTIONS. I.e. to start an image transfer on a specific host: 1. Sdk user specifies a host on ImageTransfer in order to use unix socket on that host. 2. On the host, client issue OPTIONS request on the transfer_url. 3. If imageio supports unix socket, we return the unix socket in the response: { “features”: [“zero”, “flush”], “unix_socket”: “\0/org/ovirt/imageio” } 4. If the local imageio daemon supports unix socket, the client will send the data over the unix socket, otherwise fallback to the transfer_url.
This feature is needed for: - virt-v2v for importing vm disks. Will allow improving more disks in the same time using less resources. - backup vendors - will improve backup solution scalability. The flow we want to support: 1. user program starts on a some host in a data center A user system will select hosts for image transfers using ovirt API. The transfers will be distributed over all hosts in a data center to distribute the I/O. This is similar to the way we distribute storage jobs since 4.1. 2. user program finds the host uuid Currently the host uuid used in engine side is not kept on the host, but we can find it using the host hw_uuid at /etc/vdsm/vdsm.id. Example call to find host: with open("/etc/vdsm/vdsm.id") as f: vdsm_id = f.readline().strip() hosts_service = connection.system_service().hosts_service() hosts = hosts_service.list( search='vds_unique_id=%s' % vdsm_id, case_sensitive=False, ) Depends on https://gerrit.ovirt.org/#/c/92025/ 3. user program starts an image transfer using the found host uuid This is already supported by specifying a host. Here is an example call: transfers_service = system_service.image_transfers_service() transfer = transfers_service.add( types.ImageTransfer( disk=types.Disk(id='123'), host=types.Host(id=hosts[0].id) ) ) 4. user program find the unix socket address using OPTIONS on the transfer_url The first request to the server should be OPTIONS /images/ticket-uuid anyway, to find the features supported by the server (e.g. "zero"). This will also expose the unix socket address. Example request: OPTIONS /images/ticket-uuid .. { "features": ["zero", "flush"], "unix_socket": "\0/org/ovirt/imageio" } 5. user program transfer the data using the unix socket The program need to use a modified httplib.HTTPConnection to connect using socket address. Then it can use standard HTTPConnection apis to perform the transfers. We have example implementation here: https://github.com/oVirt/ovirt-imageio/blob/1ef5af7e47495de889142960bafa2ca089d81de3/daemon/ovirt_imageio_daemon/uhttp.py#L36 Users can also use this pypi package: https://pypi.org/project/uhttplib/
Daniel cna you please cross check if all patches have been merged in the version being released in 4.2.5 and move to QE accordingly?
(In reply to Sandro Bonazzola from comment #4) > Daniel cna you please cross check if all patches have been merged in the > version being released in 4.2.5 and move to QE accordingly? sure, moved to ON_QA
As I know the Unix Socket support is not part of V2V v1 GA, Brett told me it is postponed to V2V v1.1 Nir is it supported end to end with virt v2v? If yes please suggest which versions? If this can be tested without virt v2v please suggest how. Please suggest validation instructions for both cases.
(In reply to Daniel Gur from comment #6) > As I know the Unix Socket support is not part of V2V v1 GA, Brett told me it > is postponed to V2V v1.1 > > Nir is it supported end to end with virt v2v? > If yes please suggest which versions? virt-v2v supports this since this patch: b7a2e6270 v2v: -o rhv-upload: Use Unix domain socket to access imageio (RHBZ#1588088). Which was released in v1.39.6. Some issues that should not affect the RHV use case were fixed later in 1.39.7: 4ed1bc5a7 v2v: rhv plugin - find suitable host (RHBZ#1596810) (RHBZ#1596851) 891b5a0ec v2v: rhv-upload-plugin: Optimize only direct upload f5442d2f0 v2v: rhv-upload-plugin: Improve error handling d5f36bacf v2v: rhv-upload-plugin: Remove unneeded auth And there are these pending changes which are not included in any release yet but should not block testing: 23b62f391 v2v: rhv plugin - case-sensitive search queries 2547df8a0 v2v: rhv plugin - fix DC search string Downstream virt-v2v supports unix socket so using it successfully verify this bug. > If this can be tested without virt v2v please suggest how. > > Please suggest validation instructions for both cases. You can test this without virt-v2v using the same procedure suggested here: https://bugzilla.redhat.com/show_bug.cgi?id=1591534#c4 How to verify that upload was using unix socket: 1. virt-v2v import You should see this log when running in --verbose mode: $ grep optimizing leopard04_import/v2v-import-20180710T081*.log leopard04_import/v2v-import-20180710T081541-637.log:optimizing connection using unix socket '\x00/org/ovirt/imageio' leopard04_import/v2v-import-20180710T081542-690.log:optimizing connection using unix socket '\x00/org/ovirt/imageio' This was taken from Guy virt-v2v test results: "Performance test results - regression 10 V2V 10 VMS 1 migration plan" sent on Jul 10. 2. manual upload using imageio examples/upload script Check daemon logs - you should see this pattern: OPTIONS request issued using HTTPS: 2018-07-10 08:17:17,932 INFO (Thread-5) [web] START: [10.35.68.26] OPTIONS /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3 2018-07-10 08:17:17,933 INFO (Thread-5) [web] FINISH [10.35.68.26] OPTIONS /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3: [200] 74 [request=0.001215] We know that this is HTTPS by the client address ([10.35.68.26]). Next request using unix socket: 2018-07-10 08:17:17,943 INFO (Thread-6) [web] START: [^@] PUT /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3 2018-07-10 08:17:17,944 INFO (Thread-6) [images] Writing 52736 bytes at offset 0 flush False to /rhev/data-center/mnt/blockSD/e30bfac2-8e13-479d-8cd6-c6da5e306c4e/images/e1665094-894e-427a-b5db-41a00ec6bfb0/65ff827d-4277-4dc8-8245-5ee4f418d5c9 for ticket 98e55158-d9cd-47a3-923d-8d026e0e5bd3 2018-07-10 08:17:17,946 INFO (Thread-6) [web] FINISH [^@] PUT /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3: [200] 0 [request=0.002167, operation=0.001513, read=0.000123, write=0.000683] With unix socket client address is displayed as "[^@]". This will be improved in 1.4.3 to show "[local]". Basically I think the tests Guy did on Jul 10 verified this bug. If you want to spend more time on this, it would be nice to compare uploading same image with unix socket and without, and report the improvement.
Yes, it is verified on my tests we are using unix socket.
This bugzilla is included in oVirt 4.2.5 release, published on July 30th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.5 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.