Bug 1320606

Created attachment 1139644 [details]
log files from the host

Some logs from the host, the network configuration, and the host-deploy logs from the engine.

Created attachment 1139646 [details]
the networks

Created attachment 1139650 [details]
locked hosts

What I get when a want to do operations on the hosts, even after I rebooted it using the GUI.

Created attachment 1139651 [details]
the failed network configuration for this host

Created attachment 1139656 [details]
the engine.log

jsonrpc.Executor/0::DEBUG::2016-03-23 14:51:45,528::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with {}
jsonrpc.Executor/0::DEBUG::2016-03-23 14:51:45,529::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True
jsonrpc.Executor/1::DEBUG::2016-03-23 14:51:45,532::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with {}
jsonrpc.Executor/1::DEBUG::2016-03-23 14:51:45,532::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True

vdsm.log ends with this tight loop, while engine.log sees an network exception

2016-03-23 14:17:10,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-27) [279c2be] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Client close

Martin, could the network be flooded and disconnected due to the bug fixed by your https://gerrit.ovirt.org/#/c/54644/ ?

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

(In reply to Dan Kenigsberg from comment #6)
> jsonrpc.Executor/0::DEBUG::2016-03-23
> 14:51:45,528::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling
> 'Host.ping' in bridge with {}
> jsonrpc.Executor/0::DEBUG::2016-03-23
> 14:51:45,529::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return
> 'Host.ping' in bridge with True
> jsonrpc.Executor/1::DEBUG::2016-03-23
> 14:51:45,532::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling
> 'Host.ping' in bridge with {}
> jsonrpc.Executor/1::DEBUG::2016-03-23
> 14:51:45,532::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return
> 'Host.ping' in bridge with True
> 
> vdsm.log ends with this tight loop, while engine.log sees an network
> exception
> 
> 2016-03-23 14:17:10,080 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (default task-27)
> [279c2be] Exception:
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
> VDSGenericException: VDSNetworkException: Client close
> 
> Martin, could the network be flooded and disconnected due to the bug fixed
> by your https://gerrit.ovirt.org/#/c/54644/ ?

Hardly — this patch blocks flood instead, and it wasn't merged at the timestamp of your question or reported issues.

> > Martin, could the network be flooded and disconnected due to the bug fixed
> > by your https://gerrit.ovirt.org/#/c/54644/ ?
> 
> Hardly — this patch blocks flood instead, and it wasn't merged at the
> timestamp of your question or reported issues.

Martin, let me rephrase my question. Could it be that the connection is broken due to the ping flood, which is solved by your patch?

Sorry, I was reading too quickly. I'm not sure, but it might be the case. In engine.log there is:

org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
	at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:157) [vdsm-jsonrpc-java-client.jar:]

so the vdsm-jsonrpc is trying to establish connection, but it fails. It might be the case, that vds is clogged by that ping DOS. But I havent notice described behavior even before fixing this bug — even then I was able to add host despite of flood.

is this a duplicate: BZ 1329317 ?

Add host stuck in installing state with ping flood issue, i failed with this symptoms 1 time of a 3 attempts. Not sure if this can be verified, attaching logs.

Tested on 3.6.6-0.1.el6 and vdsm-4.17.27-0.el7ev.noarch

Created attachment 1151744 [details]
New Logs

Pings in your log are received every 1.5 seconds. The tight loop we've seen before is verified.

However, bug 1329317 is still unsolved. Which version of vdsm-jsonrpc-java have you used?

jsonrpc.Executor/0::DEBUG::2016-04-28 09:10:13,641::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with {}
jsonrpc.Executor/0::DEBUG::2016-04-28 09:10:13,641::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True
jsonrpc.Executor/1::DEBUG::2016-04-28 09:10:15,146::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with {}
jsonrpc.Executor/1::DEBUG::2016-04-28 09:10:15,147::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True
jsonrpc.Executor/2::DEBUG::2016-04-28 09:10:16,652::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with {}
jsonrpc.Executor/2::DEBUG::2016-04-28 09:10:16,652::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True
jsonrpc.Executor/3::DEBUG::2016-04-28 09:10:17,953::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.ping' in bridge with {}
jsonrpc.Executor/3::DEBUG::2016-04-28 09:10:17,954::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.ping' in bridge with True

vdsm-jsonrpc-java-1.1.9-1.el6ev.noarch

Verified on - 3.6.6-0.1.el6 and vdsm-4.17.27-0.el7ev.noarch

verification of  bug 1329317 must take place with vdsm-jsonrpc-java >= 1.1.10 and ovirt-engine >= 3.6.0_alpha1-2572-g16e91cf.