Bug 1585675 - [RFE] imageio unix socket support
Summary: [RFE] imageio unix socket support
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-imageio
Classification: oVirt
Component: General
Version: ---
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.2.5
: ---
Assignee: Daniel Erez
QA Contact: guy chen
URL:
Whiteboard:
Depends On:
Blocks: 1588088
TreeView+ depends on / blocked
 
Reported: 2018-06-04 11:59 UTC by Daniel Erez
Modified: 2018-08-28 11:36 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Add UNIX socket support. Full documentation is available at: http://ovirt.github.io/ovirt-imageio/unix-socket.html
Clone Of:
Environment:
Last Closed: 2018-07-31 15:32:29 UTC
oVirt Team: Scale
rule-engine: ovirt-4.2?
ebenahar: testing_plan_complete?
rule-engine: planning_ack?
rule-engine: devel_ack+
ebenahar: testing_ack+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 91443 master MERGED daemon: Serve /images/ locally 2018-06-12 16:56:17 UTC
oVirt gerrit 92025 master MERGED core: support hw_id in vds search 2018-06-07 19:09:57 UTC
oVirt gerrit 92032 master MERGED images: Report unix socket address in OPTIONS 2018-06-12 16:56:15 UTC
oVirt gerrit 92071 master MERGED examples: Use unix socket if possible 2018-06-12 16:56:20 UTC
oVirt gerrit 92158 ovirt-engine-4.2 MERGED core: support hw_id in vds search 2018-06-13 07:28:27 UTC

Description Daniel Erez 2018-06-04 11:59:55 UTC
Testing upload shows that reading from SSL socket is our bottleneck with fast storage. Using http over unix socket, we can eliminate this overhead when the upload is performed locally.

Required changes:
* Engine - expose a unix socket attribute in ImageTransfer entity (when the host supports this feature). 
* vdsm - return imageio info on getCaps.
* imageio-daemon - service ‘/info’ over https and unix control socket.

Comment 1 Yaniv Kaul 2018-06-05 07:29:54 UTC
severity?

Comment 2 Daniel Erez 2018-06-06 12:52:04 UTC
We will start by exposing the unix socket only from OPTIONS.
I.e. to start an image transfer on a specific host:
1. Sdk user specifies a host on ImageTransfer in order to use unix socket on that host.
2. On the host, client issue OPTIONS request on the transfer_url.
3. If imageio supports unix socket, we return the unix socket in the response:
   {
       “features”: [“zero”, “flush”],
       “unix_socket”: “\0/org/ovirt/imageio”
   }
4. If the local imageio daemon supports unix socket, the client will send the data over the unix socket, otherwise fallback to the transfer_url.

Comment 3 Nir Soffer 2018-06-07 11:30:09 UTC
This feature is needed for:
- virt-v2v for importing vm disks. Will allow improving more disks in the same
  time using less resources. 
- backup vendors - will improve backup solution scalability. 


The flow we want to support:

1. user program starts on a some host in a data center

A user system will select hosts for image transfers using ovirt API. The transfers
will be distributed over all hosts in a data center to distribute the I/O. This is
similar to the way we distribute storage jobs since 4.1.


2. user program finds the host uuid

Currently the host uuid used in engine side is not kept on the host, but 
we can find it using the host hw_uuid at /etc/vdsm/vdsm.id.

Example call to find host:

    with open("/etc/vdsm/vdsm.id") as f:
        vdsm_id = f.readline().strip()

    hosts_service = connection.system_service().hosts_service()
    hosts = hosts_service.list(
        search='vds_unique_id=%s' % vdsm_id,
        case_sensitive=False,
    )

Depends on https://gerrit.ovirt.org/#/c/92025/


3. user program starts an image transfer using the found host uuid

This is already supported by specifying a host. Here is an example call:

transfers_service = system_service.image_transfers_service()
transfer = transfers_service.add(
    types.ImageTransfer(
        disk=types.Disk(id='123'),
        host=types.Host(id=hosts[0].id)
   )
)


4. user program find the unix socket address using OPTIONS on the transfer_url

The first request to the server should be OPTIONS /images/ticket-uuid anyway, to 
find the features supported by the server (e.g. "zero"). This will also expose
the unix socket address.

Example request:

    OPTIONS /images/ticket-uuid
    ..
    {
        "features": ["zero", "flush"],
        "unix_socket": "\0/org/ovirt/imageio"
    }

5. user program transfer the data using the unix socket

The program need to use a modified httplib.HTTPConnection to connect using socket
address. Then it can use standard HTTPConnection apis to perform the transfers.

We have example implementation here:
https://github.com/oVirt/ovirt-imageio/blob/1ef5af7e47495de889142960bafa2ca089d81de3/daemon/ovirt_imageio_daemon/uhttp.py#L36

Users can also use this pypi package:
https://pypi.org/project/uhttplib/

Comment 4 Sandro Bonazzola 2018-07-24 16:11:49 UTC
Daniel cna you please cross check if all patches have been merged in the version being released in 4.2.5 and move to QE accordingly?

Comment 5 Daniel Erez 2018-07-24 16:23:13 UTC
(In reply to Sandro Bonazzola from comment #4)
> Daniel cna you please cross check if all patches have been merged in the
> version being released in 4.2.5 and move to QE accordingly?

sure, moved to ON_QA

Comment 6 Daniel Gur 2018-07-26 12:29:35 UTC
As I know the Unix Socket support is not part of V2V v1 GA, Brett told me it is postponed to V2V v1.1

Nir is it supported end to end with virt v2v? 
If yes please suggest which versions?

If this can be tested without virt v2v please suggest how.

Please suggest validation instructions for both cases.

Comment 7 Nir Soffer 2018-07-26 13:14:38 UTC
(In reply to Daniel Gur from comment #6)
> As I know the Unix Socket support is not part of V2V v1 GA, Brett told me it
> is postponed to V2V v1.1
> 
> Nir is it supported end to end with virt v2v? 
> If yes please suggest which versions?

virt-v2v supports this since this patch:
b7a2e6270 v2v: -o rhv-upload: Use Unix domain socket to access imageio (RHBZ#1588088).

Which was released in v1.39.6.

Some issues that should not affect the RHV use case were fixed later in 1.39.7:
4ed1bc5a7 v2v: rhv plugin - find suitable host (RHBZ#1596810) (RHBZ#1596851)
891b5a0ec v2v: rhv-upload-plugin: Optimize only direct upload
f5442d2f0 v2v: rhv-upload-plugin: Improve error handling
d5f36bacf v2v: rhv-upload-plugin: Remove unneeded auth

And there are these pending changes which are not included in any release yet
but should not block testing:
23b62f391 v2v: rhv plugin - case-sensitive search queries
2547df8a0 v2v: rhv plugin - fix DC search string

Downstream virt-v2v supports unix socket so using it successfully verify this bug.

> If this can be tested without virt v2v please suggest how.
> 
> Please suggest validation instructions for both cases.

You can test this without virt-v2v using the same procedure suggested here:
https://bugzilla.redhat.com/show_bug.cgi?id=1591534#c4

How to verify that upload was using unix socket:

1. virt-v2v import

You should see this log when running in --verbose mode:

$ grep optimizing leopard04_import/v2v-import-20180710T081*.log
leopard04_import/v2v-import-20180710T081541-637.log:optimizing connection using unix socket '\x00/org/ovirt/imageio'
leopard04_import/v2v-import-20180710T081542-690.log:optimizing connection using unix socket '\x00/org/ovirt/imageio'

This was taken from Guy virt-v2v test results:
"Performance test results - regression 10 V2V 10 VMS 1 migration plan"
sent on Jul 10.

2. manual upload using imageio examples/upload script

Check daemon logs - you should see this pattern:

OPTIONS request issued using HTTPS:

2018-07-10 08:17:17,932 INFO    (Thread-5) [web] START: [10.35.68.26] OPTIONS /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3
2018-07-10 08:17:17,933 INFO    (Thread-5) [web] FINISH [10.35.68.26] OPTIONS /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3: [200] 74 [request=0.001215]

We know that this is HTTPS by the client address ([10.35.68.26]).

Next request using unix socket:

2018-07-10 08:17:17,943 INFO    (Thread-6) [web] START: [^@] PUT /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3
2018-07-10 08:17:17,944 INFO    (Thread-6) [images] Writing 52736 bytes at offset 0 flush False to /rhev/data-center/mnt/blockSD/e30bfac2-8e13-479d-8cd6-c6da5e306c4e/images/e1665094-894e-427a-b5db-41a00ec6bfb0/65ff827d-4277-4dc8-8245-5ee4f418d5c9 for ticket 98e55158-d9cd-47a3-923d-8d026e0e5bd3
2018-07-10 08:17:17,946 INFO    (Thread-6) [web] FINISH [^@] PUT /images/98e55158-d9cd-47a3-923d-8d026e0e5bd3: [200] 0 [request=0.002167, operation=0.001513, read=0.000123, write=0.000683]

With unix socket client address is displayed as "[^@]". This will be improved 
in 1.4.3 to show "[local]".

Basically I think the tests Guy did on Jul 10 verified this bug.

If you want to spend more time on this, it would be nice to compare uploading same image with unix socket
and without, and report the improvement.

Comment 8 guy chen 2018-07-31 12:36:35 UTC
Yes, it is verified on my tests we are using unix socket.

Comment 9 Sandro Bonazzola 2018-07-31 15:32:29 UTC
This bugzilla is included in oVirt 4.2.5 release, published on July 30th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.5 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.