Bug 1331396 - Setup Networks cause ClosedChannelException
Summary: Setup Networks cause ClosedChannelException
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm-jsonrpc-java
Version: 3.6.4
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-3.6.6
: 3.6.6
Assignee: Piotr Kliczewski
QA Contact: Pavol Brilla
URL:
Whiteboard:
Depends On: 1323465
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-28 12:55 UTC by Pavel Zhukov
Modified: 2019-10-10 12:00 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1323465
Environment:
Last Closed: 2016-05-27 19:37:43 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 55681 0 None None None 2016-04-28 12:55:38 UTC
oVirt gerrit 55855 0 None None None 2016-04-28 12:55:38 UTC
oVirt gerrit 55872 0 master ABANDONED jsonrpc: make sure not to block when processing i/o 2016-05-04 07:39:45 UTC
oVirt gerrit 55940 0 None None None 2016-04-28 12:55:38 UTC
oVirt gerrit 55962 0 None None None 2016-04-28 12:55:38 UTC
oVirt gerrit 55968 0 None None None 2016-04-28 12:55:38 UTC
oVirt gerrit 55969 0 None None None 2016-04-28 12:55:38 UTC

Description Pavel Zhukov 2016-04-28 12:55:39 UTC
+++ This bug was initially created as a clone of Bug #1323465 +++

Description of problem:
Setup Networks operation hangs out for ever and can't be rolled back when changing the ovirtmgmt's interface.
To recover from it we will need to restart the ovirt-engine service.

When changing the ovritmgmt interface via setup networks, operation can't be finished and there is no roll back to last known configuration.

Sometimes operation failing with communication error with engine and host, host moving to connecting state. In the UI operation seems to be failing, although  the change is took place on the host.

Sometimes operation succeeded without any errors.

Sometimes operation failing with connection to peer error and then host rolls back to last network configuration after a very long time out.

Version-Release number of selected component (if applicable):
3.6.5-0.1.el6.noarch

Steps to Reproduce:
1. Move the ovirtmgmt network to other interface on host via setup networks dialog


Actual results:
- Setup Networks operation hangs out for ever and can't be rolled back. Restarting the ovirt-engine service will recover the engine. 

- Operation failed with communication error in the UI engine, but the change took place on host.

- Operation failed with connection to peer error in the UI engine. The change didn't took place on the host and host rolled back to latest network configurations as expected.  

- Operation succeeded. 

Expected results:
In case when we should receive the same IP on the other interface, operation should succeed. 

In case we should receive different IP, we should fail and host should roll back to last configurations. 

Setup Networks operation shouldn't hang out fore ever without failing. In case of any communication or connection issues between host and engine, host should be rolled back to last known network configuration.

--- Additional comment from Michael Burman on 2016-04-03 06:45:42 EDT ---

Please note, that when engine stuck in such state as described above(hanging for ever with out any time out or failure), we can't approve any other setup networks operations --> "cannot setup networks. another setup network process in progress  on the host". To recover form it we need to restart the engine.

--- Additional comment from Piotr Kliczewski on 2016-04-04 09:57:29 EDT ---

During setupNetworks we get:

2016-04-04 16:20:57,184 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) [rt.jar:1.8.0_71]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) [rt.jar:1.8.0_71]
        at org.ovirt.vdsm.jsonrpc.client.reactors.SSLEngineNioHelper.read(SSLEngineNioHelper.java:50) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.read(SSLClient.java:91) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:103) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:204) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:125) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.processChannels(Reactor.java:89) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:65) [vdsm-jsonrpc-java-client.jar:]


which causes:

2016-04-04 16:20:57,249 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Internal server error: java.lang.NullPointerException

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-04 10:52:39 EDT ---

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-04 10:52:39 EDT ---

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 5 Martin Perina 2016-05-09 13:15:30 UTC
After discussion with Pavel I've changed the Version to 3.6.4 according to customer ticket. Moving status to ON_QA, same as orignal bug

Comment 6 Pavol Brilla 2016-05-11 16:00:32 UTC
verified on vdsm-jsonrpc-4.17.28-0.el7ev.noarch

Comment 7 Pavel Zhukov 2016-05-23 07:14:53 UTC
(In reply to Pavol Brilla from comment #6)
> verified on vdsm-jsonrpc-4.17.28-0.el7ev.noarch

Can you elaborate please?
The bug was in vdsm-jsonrpc-java. How did you manage to verify it using different package?

Comment 8 Martin Perina 2016-05-23 08:06:44 UTC
(In reply to Pavel Zhukov from comment #7)
> (In reply to Pavol Brilla from comment #6)
> > verified on vdsm-jsonrpc-4.17.28-0.el7ev.noarch
> 
> Can you elaborate please?
> The bug was in vdsm-jsonrpc-java. How did you manage to verify it using
> different package?

Pavel, the whole SetupNetworks/jsonrpc/ping flood problem was not a single issue, but several issues at once which broke host installation. So there were multiple patches on engine, vdsm-jsonrpc-java and vdsm sides. The important thing is that both 3.6.6 and 4.0.0beta2 don't suffer from the issue.

Comment 9 Martin Perina 2016-05-27 19:37:43 UTC
RHEV 3.6.6 has been released, closing.


Note You need to log in before you can comment on or make changes to this bug.