Bug 1154398

Created attachment 948265 [details]
logs

It takes time for vdsm to start and the engine assumes that when host install is done that the vdsm is fully functional but the port is not opened for a while.

All the connection attempts were successful. The network was reconfigured and the communication was up and running each time. Vdsm and engine log confirms it.

There were some other issues like:

 java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:/ENGINEDataSource

Please give more details about the nature of failure because it looks like the connection was always there according to the logs.

Created attachment 948560 [details]
reproduced,fresh setup logs

(In reply to Piotr Kliczewski from comment #2)
> It takes time for vdsm to start and the engine assumes that when host
> install is done that the vdsm is fully functional but the port is not opened
> for a while.
> 
> All the connection attempts were successful. The network was reconfigured
> and the communication was up and running each time. Vdsm and engine log
> confirms it.
> 
> There were some other issues like:
> 
>  java.sql.SQLException: javax.resource.ResourceException: IJ000451: The
> connection manager is shutdown: java:/ENGINEDataSource
> 
> Please give more details about the nature of failure because it looks like
> the connection was always there according to the logs.

yes, I ran some tests and explored this issue,it seems that the bug happens on hosts which do not login their iscsiadm sessions upon restart.

for example host-vdsb doesn't prompt this bug and the result of iscsiadm -m session upon restart is:

root@camel-vdsb ~ # iscsiadm -m session
tcp: [1] 10.35.160.104:3260,1 iqn.1992-04.com.emc:cx.ckm00121000438.a6
tcp: [2] 10.35.160.106:3260,3 iqn.1992-04.com.emc:cx.ckm00121000438.b6
tcp: [3] 10.35.160.107:3260,4 iqn.1992-04.com.emc:cx.ckm00121000438.b7
tcp: [4] 10.35.160.105:3260,2 iqn.1992-04.com.emc:cx.ckm00121000438.a7

camel-vdsc which gives the error:

root@camel-vdsc ~ # iscsiadm -m session                   
iscsiadm: No active sessions.

                 **note** 
no add domain operation is initiated at all, The fail happens during add host operation on a fresh engine.

I searched bugzilla and found this, BZ #1056948, I have checked camel-vdsc iscsi.conf file, and the command appears as it should:

# scripts run, set the iscsid startup command here. This
iscsid.startup = /etc/rc.d/init.d/iscsid force-start
# iscsid.startup = /sbin/iscsid

In newly attached logs I can see:

Status of host camel-vdsc was set to Up

I agree that there are some exceptions in the log but there are situations that vdsm is really slow to start. I am working on reducing verbosity of the logs so
next time it won't be so confusing.

I do not see anything in the logs that makes add domain operation not initialized related to communication.

I stand corrected,The iscsi issue is a different bug,this bug happent on vdsm vt5 hosts regardless to iscsi

It looks like this is log verbosity issue. We need to reduce amount of information that is logged during connecting phase.

ok vdsm-4.16.8.1-5.el6ev.x86_64

there are no stacktraces anymore like

WARN  [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) Retry failed: java.net.ConnectException: Connection refused

rhev 3.5.0 was released. closing.