Bug 1801710

Summary: ovirt-imageio service not available on engine host after clean reprovision meaning upload/download operations from UI do not work
Product: [oVirt] ovirt-engine Reporter: Yosi Ben Shimon <ybenshim>
Component: BLL.StorageAssignee: Vojtech Juranek <vjuranek>
Status: CLOSED CURRENTRELEASE QA Contact: Ilan Zuckerman <izuckerm>
Severity: high Docs Contact:
Priority: high    
Version: 4.4.0CC: aefrat, aoconnor, bugs, jzmeskal, lsvaty, michal.skrivanek, nsoffer, ralonso, sbonazzo, tnisan
Target Milestone: ovirt-4.4.0Keywords: Automation, AutomationBlocker, Regression, TestBlocker
Target Release: 4.4.0Flags: pm-rhel: ovirt-4.4+
aoconnor: blocker+
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.4.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-20 20:01:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1862059    
Bug Blocks:    

Description Yosi Ben Shimon 2020-02-11 13:54:40 UTC
Description of problem:
After new reprovision of rhv 4.4, the ovirt-imageio-proxy service is not running.
# rpm -qa | grep imageio-proxy 
returns nothing

Using automation tests, the test failed with the next errors in the engine log:

2020-02-05 01:31:09,660+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-129) [5558cd2d-29c0-4f40-b9d3-42ba2a740ea8] Failed to add image ticket to ovirt-imageio-proxy: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
	at java.base/java.net.Socket.connect(Socket.java:591)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)
	at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)


..... long stack trace


2020-02-05 01:31:09,661+02 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-129) [5558cd2d-29c0-4f40-b9d3-42ba2a740ea8] Failed to add image ticket to ovirt-imageio-proxy
2020-02-05 01:31:09,689+02 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-129) [5558cd2d-29c0-4f40-b9d3-42ba2a740ea8] EVENT_ID: TRANSFER_FAILED_TO_ADD_TICKET_TO_PROXY(1,075), Failed to add image ticket to ovirt-imageio-proxy - only transferring directly to ovirt-imageio-daemon is applicable (use 'transfer_url').

Apperantly, the ovirt-imageio-proxy was not installed.

Version-Release number of selected component (if applicable):
ovirt-engine-4.4.0-0.19.master.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. install/update rhv environment
2. execute "rpm -qa | grep imageio"
3.

Actual results:
The ovirt-imageio-proxy is not installed

Expected results:
The ovirt-imageio-proxy should be installed and running

Additional info:

Comment 1 Nir Soffer 2020-02-11 14:10:10 UTC
Engine 4.4. is in the middle of migration to python 3 and imageio proxy
package may not be available until we finish this migration.

The proxy is used only for upload and download from UI, so it should not affect
upload/download tests which should use transfer_url. If you have tests using
only proxy_url they should be changed to test both transfer_url and proxy_url.

Tests using proxy_url should be skipped until we finish the migration to python 3.

Comment 4 Tal Nisan 2020-02-17 15:25:34 UTC
Sandro, can you have a look if the package was dropped by mistake?

Comment 5 Sandro Bonazzola 2020-02-28 12:25:58 UTC
(In reply to Tal Nisan from comment #4)
> Sandro, can you have a look if the package was dropped by mistake?

package has not been built for el8 and is not included in el8 repos.
You can take it from el7: https://resources.ovirt.org/repos/ovirt/tested/master/rpm/el7/noarch/ovirt-imageio-proxy-1.6.3-0.el7.noarch.rpm
but can't be automatically installed unless built for el8.

Comment 6 Michal Skrivanek 2020-03-03 13:45:43 UTC
we won't get it in some time, retargetting for 4.4.2 for now

Comment 8 Jan Zmeskal 2020-03-23 12:48:16 UTC
Just want to expand on what Nir wrote in comment #1:
The proxy is used only for upload and download from UI, so it should not affect
upload/download tests which should use transfer_url. If you have tests using
only proxy_url they should be changed to test both transfer_url and proxy_url.

This is not entirely true, since Ansible ovirt_disk module uses url_proxy as well. See here: https://github.com/ansible/ansible/blob/4e8b240b8b74db2453851f12ec82c7f782289187/lib/ansible/modules/cloud/ovirt/ovirt_disk.py#L377

This affects ovirt_disk module in a way described here: https://bugzilla.redhat.com/show_bug.cgi?id=1802661#c7

Comment 9 Nir Soffer 2020-03-23 13:02:14 UTC
(In reply to Jan Zmeskal from comment #8)
> Just want to expand on what Nir wrote in comment #1:
> The proxy is used only for upload and download from UI, so it should not
> affect
> upload/download tests which should use transfer_url. If you have tests using
> only proxy_url they should be changed to test both transfer_url and
> proxy_url.
> 
> This is not entirely true, since Ansible ovirt_disk module uses url_proxy as
> well. See here:
> https://github.com/ansible/ansible/blob/
> 4e8b240b8b74db2453851f12ec82c7f782289187/lib/ansible/modules/cloud/ovirt/
> ovirt_disk.py#L377

This is ovirt_disk ansible module bug, using outdated code. It should use the code in:
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/upload_disk.py

It can support using a proxy for cases when you don't have access to host, but
the recommended and supported way to upload is via the host directly.

> This affects ovirt_disk module in a way described here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1802661#c7

Should be fixed in that package.

Comment 10 Jan Zmeskal 2020-03-23 13:57:38 UTC
Hi Nir, is that bug tracked somewhere, please?

Comment 11 Nir Soffer 2020-03-23 14:02:26 UTC
(In reply to Jan Zmeskal from comment #10)
> Hi Nir, is that bug tracked somewhere, please?

I don't know about a bug, please file one.

Comment 12 Jan Zmeskal 2020-03-23 14:40:02 UTC
> I don't know about a bug, please file one.

The Ansible ovirt_disk issue is now being tracked here: https://github.com/ansible/ansible/issues/68403
Thank you for the explanation, Nir!

Comment 13 Lukas Svaty 2020-03-24 12:50:59 UTC
due to Regression keyword targeting to 4.4.0

Comment 16 Nir Soffer 2020-04-19 13:59:59 UTC
Moving back to 4.4.0, since the patches looks ready.

Comment 19 Ilan Zuckerman 2020-05-13 12:28:31 UTC
package ovirt-imageio-proxy was completely remove and daemon, which previously run only on the host, can now server also as a proxy, i.e. daemon now run host as well as on engine. The service is called ovirt-imageio.

In terms of packages, we have two packages:
* ovirt-imageio-common - basically imageio library, containing most of thecode
* ovirt-imageio-deamon - contains daemon script, service file etc. Requires ovirt-imageio-common.

Verified on:
rhv-release-4.4.0-35-001.noarch
ovirt-imageio-daemon-2.0.5-0.el8ev.x86_64
ovirt-imageio-common-2.0.5-0.el8ev.x86_64

Service is up and running:

[root@storage-ge-05 ~]#  systemctl status ovirt-imageio
● ovirt-imageio.service - oVirt ImageIO Daemon
   Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-05-12 17:08:37 IDT; 17h ago
 Main PID: 1246057 (ovirt-imageio)
    Tasks: 3 (limit: 101312)
   Memory: 12.1M
   CGroup: /system.slice/ovirt-imageio.service
           └─1246057 /usr/libexec/platform-python -s /usr/bin/ovirt-imageio


I uploaded a disk image from UI while watching /var/log/ovirt-imageio/daemon.log both on engine and vdsm host.
No ERRORS / WARNS were seen, the process completed successfully.

Comment 20 Sandro Bonazzola 2020-05-20 20:01:18 UTC
This bugzilla is included in oVirt 4.4.0 release, published on May 20th 2020.

Since the problem described in this bug report should be
resolved in oVirt 4.4.0 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 21 Ilan Zuckerman 2020-07-30 10:18:18 UTC
This bug depends on libvirt bz
https://bugzilla.redhat.com/show_bug.cgi?id=1862059