Bug 1379130

Summary: Unexpected client exception when glance server is not reachable
Product: [oVirt] ovirt-engine Reporter: Yaniv Kaul <ykaul>
Component: BLL.StorageAssignee: Benny Zlotnik <bzlotnik>
Status: CLOSED CURRENTRELEASE QA Contact: Elad <ebenahar>
Severity: medium Docs Contact:
Priority: unspecified    
Version: ---CC: bugs, juan.hernandez, ratamir, tnisan
Target Milestone: ovirt-4.1.1Keywords: AutomationBlocker
Target Release: 4.1.1Flags: rule-engine: ovirt-4.1+
rule-engine: planning_ack+
rule-engine: devel_ack+
ratamir: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-21 09:36:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yaniv Kaul 2016-09-25 09:48:03 UTC
Description of problem:
When setting glance.ovirt.org address to 127.0.0.1 and trying to reach it and list images from it, I'm getting the following stack:
  File "/home/ykaul/ovirt-system-tests/basic_suite_master/test-scenarios/002_bootstrap.py", line 392, in list_glance_images
    all_images = glance_provider.images.list()
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 26117, in list
    headers={}
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 46, in get
    return self.request(method='GET', url=url, headers=headers, cls=cls)
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request
    persistent_auth=self.__persistent_auth
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request
    persistent_auth)
  File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request
    raise errors.RequestError(response_code, response_reason, response_body)
RequestError: 
status: 400
reason: Bad Request
detail: unexpected client exception


I'd expect that exception to be caught and handled by the SDK and return an empty list, since this is not a server error - we never reached the server.



Version-Release number of selected component (if applicable):
ovirt-engine-sdk-python-3.6.9.1-1.fc24.noarch

How reproducible:
Always

Comment 1 Yaniv Kaul 2016-10-25 09:46:29 UTC
Adding AutomationBlocker flag, as it affects CI (see http://jenkins.ovirt.org/job/ovirt_master_system-tests/692/testReport/junit/(root)/002_bootstrap/list_glance_images/ for example).

Comment 2 Juan Hernández 2016-10-25 10:10:35 UTC
The relevant error message from engine.log is the following:

Caused by: java.net.UnknownHostException: glance.ovirt.org: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) [rt.jar:1.8.0_111]
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) [rt.jar:1.8.0_111]
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) [rt.jar:1.8.0_111]
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276) [rt.jar:1.8.0_111]
	at java.net.InetAddress.getAllByName(InetAddress.java:1192) [rt.jar:1.8.0_111]
	at java.net.InetAddress.getAllByName(InetAddress.java:1126) [rt.jar:1.8.0_111]

This happens when the API queries the backend, also from server.log:

2016-10-22 11:56:56,704 ERROR [org.ovirt.engine.core.bll.storage.repoimage.GetImagesListQuery] (default task-7) [] Exception: java.lang.RuntimeException: Unexpected client exception
	at com.woorea.openstack.connector.RESTEasyConnector.request(RESTEasyConnector.java:107) [resteasy-connector.jar:]
	at com.woorea.openstack.base.client.OpenStackClient.request(OpenStackClient.java:51) [openstack-client.jar:]
	at com.woorea.openstack.base.client.OpenStackClient.execute(OpenStackClient.java:66) [openstack-client.jar:]
	at com.woorea.openstack.base.client.OpenStackRequest.execute(OpenStackRequest.java:98) [openstack-client.jar:]
	at org.ovirt.engine.core.bll.provider.storage.OpenStackImageProviderProxy.getAllImagesAsRepoImages(OpenStackImageProviderProxy.java:148) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer.lambda$refreshImageDomain$1(IsoDomainListSynchronizer.java:237) [bll.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [utils.jar:]
	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
	at org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer.refreshImageDomain(IsoDomainListSynchronizer.java:232) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer.refreshRepos(IsoDomainListSynchronizer.java:208) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer.getUserRequestForStorageDomainRepoFileList(IsoDomainListSynchronizer.java:192) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.repoimage.GetImagesListQueryBase.getUserRequestForStorageDomainRepoFileList(GetImagesListQueryBase.java:42) [bll.jar:]
	at org.ovirt.engine.core.bll.storage.repoimage.GetImagesListQueryBase.executeQueryCommand(GetImagesListQueryBase.java:27) [bll.jar:]
	at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:103) [bll.jar:]
	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]

The backend should probably catch the exception and generate a meaningful error message.

There is nothing to change. I am moving to the API component for further investigation, but most probably the changes will be done in the backend.

Comment 3 Juan Hernández 2016-10-25 10:26:49 UTC
Apparently the "execute" method of the OpenStack client throws unchecked exceptions, like UnknownHostException in this case, and we don't handle them correctly. I'd suggest to wrap calls to that "execute" method with code that handles them correctly, at least the one that causes this particular error. That needs to be changed in the backend, so I'm moving to the backend component and to the storage team.

Comment 4 Elad 2017-02-15 11:32:25 UTC
Got the following error message when tried to retrieve the images list of an unreachable Glance repository (127.0.0.1):

2017-02-15 13:28:05,719+02 ERROR [org.ovirt.engine.core.bll.storage.repoimage.GetImagesListQuery] (default task-21) [dc2c439d-8ee1-4dcd-91a4-b2eafab1cfc0] Failed to retrieve image list: Connection refused (Connection refused)

Used:
rhevm-4.1.1-0.1.el7.noarch