Bug 1323465

Summary: Setup Networks cause ClosedChannelException
Product: [oVirt] ovirt-engine Reporter: Michael Burman <mburman>
Component: Backend.CoreAssignee: Piotr Kliczewski <pkliczew>
Status: CLOSED CURRENTRELEASE QA Contact: Pavol Brilla <pbrilla>
Severity: medium Docs Contact:
Priority: urgent    
Version: 3.6.0CC: bugs, eedri, gveitmic, jniederm, mgoldboi, mperina, oourfali, pkliczew, pzhukov
Target Milestone: ovirt-3.6.6Keywords: ZStream
Target Release: 3.6.6Flags: rule-engine: ovirt-3.6.z+
mgoldboi: planning_ack+
mperina: devel_ack+
pstehlik: testing_ack+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1331396 (view as bug list) Environment:
Last Closed: 2016-05-30 10:53:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1330383, 1331396    
Attachments:
Description Flags
engine log none

Description Michael Burman 2016-04-03 10:35:35 UTC
Created attachment 1142958 [details]
engine log

Description of problem:
Setup Networks operation hangs out for ever and can't be rolled back when changing the ovirtmgmt's interface.
To recover from it we will need to restart the ovirt-engine service.

When changing the ovritmgmt interface via setup networks, operation can't be finished and there is no roll back to last known configuration.

Sometimes operation failing with communication error with engine and host, host moving to connecting state. In the UI operation seems to be failing, although  the change is took place on the host.

Sometimes operation succeeded without any errors.

Sometimes operation failing with connection to peer error and then host rolls back to last network configuration after a very long time out.

Version-Release number of selected component (if applicable):
3.6.5-0.1.el6.noarch

Steps to Reproduce:
1. Move the ovirtmgmt network to other interface on host via setup networks dialog


Actual results:
- Setup Networks operation hangs out for ever and can't be rolled back. Restarting the ovirt-engine service will recover the engine. 

- Operation failed with communication error in the UI engine, but the change took place on host.

- Operation failed with connection to peer error in the UI engine. The change didn't took place on the host and host rolled back to latest network configurations as expected.  

- Operation succeeded. 

Expected results:
In case when we should receive the same IP on the other interface, operation should succeed. 

In case we should receive different IP, we should fail and host should roll back to last configurations. 

Setup Networks operation shouldn't hang out fore ever without failing. In case of any communication or connection issues between host and engine, host should be rolled back to last known network configuration.

Comment 1 Michael Burman 2016-04-03 10:45:42 UTC
Please note, that when engine stuck in such state as described above(hanging for ever with out any time out or failure), we can't approve any other setup networks operations --> "cannot setup networks. another setup network process in progress  on the host". To recover form it we need to restart the engine.

Comment 2 Piotr Kliczewski 2016-04-04 13:57:29 UTC
During setupNetworks we get:

2016-04-04 16:20:57,184 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) [rt.jar:1.8.0_71]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) [rt.jar:1.8.0_71]
        at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:50) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.read(SSLClient.java:91) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:103) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:204) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:125) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:89) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:65) [vdsm-jsonrpc-java-client.jar:]


which causes:

2016-04-04 16:20:57,249 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: java.lang.NullPointerException

Comment 3 Red Hat Bugzilla Rules Engine 2016-04-04 14:52:39 UTC
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 4 Red Hat Bugzilla Rules Engine 2016-04-04 14:52:39 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Pavol Brilla 2016-05-11 16:02:37 UTC
verified on rhevm-3.6.6.2-0.1.el6.noarch

Comment 6 Martin Perina 2016-05-20 11:53:46 UTC
*** Bug 1324155 has been marked as a duplicate of this bug. ***

Comment 7 Martin Perina 2016-06-06 07:53:13 UTC
*** Bug 1330383 has been marked as a duplicate of this bug. ***