Bug 1413932 - Engine attempts to run ConnectStorageServer when host in InstallFailed
Summary: Engine attempts to run ConnectStorageServer when host in InstallFailed
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: future
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: ovirt-4.3.0
: ---
Assignee: Idan Shaby
QA Contact: Elad
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-17 11:01 UTC by Piotr Kliczewski
Modified: 2019-04-28 08:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-10 09:10:47 UTC
oVirt Team: Storage
rule-engine: ovirt-4.3+


Attachments (Terms of Use)
engine logs (1.43 MB, application/zip)
2017-01-17 11:01 UTC, Piotr Kliczewski
no flags Details

Description Piotr Kliczewski 2017-01-17 11:01:27 UTC
Created attachment 1241672 [details]
engine logs

Engine failed to install vdsm due to broken dependencies and host was moved to installFailed:

2017-01-17 04:27:49,402-05 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-7-thread-2) [18bc6d51] START, SetVdsStatusVDSCommand(HostName = lago-basic-suite-master-host1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='bdeb76d1-30c5-4d2c-aabd-ecd6cd04522c', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 36b4cff6

but the engine attempted to call ConnectStorageServer:

DEFAULT.org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods#-9223372036854775783 as there is no unfired trigger.
2017-01-17 04:27:52,568-05 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-16) [4f11c8a9-a112-4c90-84a7-f7a0a2d229a9] START, ConnectStorageServerVDSCommand(HostName = lago-basic-suite-master-host0, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='085b3edd-639d-4bfd-81d2-11fe137e3271', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='ISCSI', connectionList='[StorageServerConnections:{id='null', connection='192.168.201.3', iqn='iqn.2014-07.org.ovirt:storage', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7d4dd3da
2017-01-17 04:27:52,570-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-16) [4f11c8a9-a112-4c90-84a7-f7a0a2d229a9] Compiled stored procedure. Call string is [{call getstorageserverconnectionextensionsbyhostidandtarget(?, ?)}]
2017-01-17 04:27:52,571-05 DEBUG [org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall] (default task-16) [4f11c8a9-a112-4c90-84a7-f7a0a2d229a9] SqlCall for procedure [GetStorageServerConnectionExtensionsByHostIdAndTarget] compiled
2017-01-17 04:27:52,579-05 INFO  [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [4f1db151] Connecting to /192.168.201.2
2017-01-17 04:27:52,596-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-16) [4f11c8a9-a112-4c90-84a7-f7a0a2d229a9] Command 'ConnectStorageServerVDSCommand(HostName = lago-basic-suite-master-host0, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='085b3edd-639d-4bfd-81d2-11fe137e3271', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='ISCSI', connectionList='[StorageServerConnections:{id='null', connection='192.168.201.3', iqn='iqn.2014-07.org.ovirt:storage', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: java.net.NoRouteToHostException: No route to host
2017-01-17 04:27:52,604-05 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-7-thread-3) [4f11c8a9-a112-4c90-84a7-f7a0a2d229a9] Host 'lago-basic-suite-master-host0' is not responding.
2017-01-17 04:27:52,596-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-16) [4f11c8a9-a112-4c90-84a7-f7a0a2d229a9] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:158) [vdsbroker.jar:]


Expected result:
Engine should not attempt to talk to the host

Comment 1 Yaniv Lavi 2017-02-07 16:20:26 UTC
Which version is this?

Comment 2 Piotr Kliczewski 2017-02-13 08:43:34 UTC
This was found on master in our CI environment.

Comment 5 Idan Shaby 2018-06-10 09:10:47 UTC
From looking at the logs (especially lago.log), it looks like after the OST tried to run add_hosts and failed, for some reason it continued to run the next tests and, among others, tried to run add_master_storage_domain (and failed), add_secondary_storage_domains (and failed) etc.
So it looks like the engine did what the OST asked him to do, which sounds fine.

Therefore, if it was a bug a year and a half ago, it was an OST bug, and not an engine one.
However, I guess that we should have seen many more like it if this bug still existed in OST.

Anyway, closing as not a bug.
If it's reproduced again, please reopen.


Note You need to log in before you can comment on or make changes to this bug.