Bug 1305876 - oVirt 3.6 won't talk to VDSM-Fake host on plaintext connection [NEEDINFO]
oVirt 3.6 won't talk to VDSM-Fake host on plaintext connection
Product: vdsm-jsonrpc-java
Classification: oVirt
Component: Core (Show other bugs)
x86_64 Linux
unspecified Severity unspecified (vote)
: ---
: ---
Assigned To: Piotr Kliczewski
Pavel Stehlik
Depends On:
  Show dependency treegraph
Reported: 2016-02-09 08:21 EST by nicolas
Modified: 2016-03-06 07:55 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-02-18 08:15:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
pkliczew: needinfo?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?

Attachments (Terms of Use)
ovirt-engine log when activating a fake host (20.54 KB, text/plain)
2016-02-09 08:21 EST, nicolas
no flags Details
Errors on ovirt engine (14.31 KB, text/plain)
2016-02-10 13:39 EST, nicolas
no flags Details
Network traffic between engine and fake host (2.53 KB, text/plain)
2016-02-10 13:40 EST, nicolas
no flags Details

  None (edit)
Description nicolas 2016-02-09 08:21:51 EST
Created attachment 1122412 [details]
ovirt-engine log when activating a fake host

Description of problem:

vdsm-fake has been cloned from Git repo and replaced version of org.ovirt.vdsm-jsonrpc-java to be 1.1.7 in pom.xml.

Build works, 'mvn jetty:run' starts correctly.

This statement is run in the DB: update vdc_options set option_value='false' where option_name='InstallVds'

Then, ovirt-engine is restarted.

When adding the host via REST as described here: http://www.ovirt.org/REST_API_Using_BASH_Automation, the host is indeed added but on activation time, the following error is shown:

Message: VDSM fakehost.mydomain.com command failed: Unrecognized SSL message, plaintext connection?

Version-Release number of selected component (if applicable):

oVirt version:
vdsm-jsonrpc-java: 1.1.7 (pom.xml)

Additional info:

I don't know if this should be fixed on ovirt-engine or vdsm-jsonrpc-java side, sorry if wrongly classified.

I'm attaching the log of ovirt-engine once the host is activated.
Comment 1 Piotr Kliczewski 2016-02-09 09:44:13 EST
What were the steps you performed on both ends to disable ssl?
Comment 2 nicolas 2016-02-09 11:16:26 EST
Seems that one of the following commands were not run correctly:

UPDATE vdc_options set option_value = 'false' WHERE option_name = 'SSLEnabled';
UPDATE vdc_options set option_value = 'false' WHERE option_name = 'EncryptHostCommunication';

Because now, after running them again restarting ovirt-engine the error stopped showing up.

I'm getting a different error now but not related to the communication between the engine and the fake VDSM, so you can close this one.

Comment 3 Piotr Kliczewski 2016-02-09 11:22:35 EST
How did you disable ssl on fake vdsm side and what kind of error are you getting now?
Comment 4 nicolas 2016-02-09 11:30:00 EST
I wasn't able to find an option for disabling SSL on the fake VDSM side, I reviewed the web.xml file but I wasn't able to find a similar option. However, disabling it on the engine side removed that error, so I assume it is talking plain by default?

The error now is:

2016-02-09 17:19:12,232 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (Stomp Reactor) [6069a834] Unable to process messages
2016-02-09 17:19:12,250 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM fakevdsm.mydomain.com command failed: Unrecognized message received 
2016-02-09 17:19:12,250 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@38065f63'
2016-02-09 17:19:12,253 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [] HostName = fakevdsm.mydomain.com
2016-02-09 17:19:12,254 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) [] Command 'GetCapabilitiesVDSCommand(HostName = fakevdsm.mydomain.com, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='30b8b9c8-d321-4941-a3a9-412984e3ac7f', vds='Host[fakevdsm.mydomain.com,30b8b9c8-d321-4941-a3a9-412984e3ac7f]'})' execution failed: VDSGenericException: VDSNetworkException: Unrecognized message received 
2016-02-09 17:19:12,254 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-54) [] Failure to refresh Vds runtime info: VDSGenericException: VDSNetworkException: Unrecognized message received 
2016-02-09 17:19:12,254 ERROR [org.ovirt.engine.core.vdsbroker.HostMonitoring] (DefaultQuartzScheduler_Worker-54) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Unrecognized message received 
	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) [vdsbroker.jar:]
	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]
	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:467) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:647) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:119) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:84) [vdsbroker.jar:]
	at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:227) [vdsbroker.jar:]
	at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source) [:1.8.0_71]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_71]
	at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_71]
	at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:]
	at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Comment 5 Piotr Kliczewski 2016-02-09 11:33:52 EST
It is required to disable ssl on both ends.

Eldad can you share where are ssl settings for fakevdsm.
Comment 6 Eldad Marciano 2016-02-10 05:32:37 EST
in webxml:
Comment 7 nicolas 2016-02-10 13:39:05 EST
Ok, right now I set the jsonSecured parameter to true, and also performed these steps:

1. On engine, I generated the p12:

    ./pki-enroll-pkcs12.sh --name=ksf --password=changeit --subject="/C=us/O=ACME/OU=People/CN=Dummy"

2. Copied the ksf.p12 file to the host.

3. Run the server with: mvn -Dfake.truststore=/etc/pki/ca-trust/extracted/java/cacerts -Dfake.keystore=/etc/tomcat/ksf.p12 jetty:run

4. Server indeed runs, but the error message is the same.

I'm attaching the error log when activating the host once added.

I'm also attaching the log of the network traffic received and sent from the fake host. As you may see the host is replying: "HTTP/1.1 400 Bad Request".
Comment 8 nicolas 2016-02-10 13:39 EST
Created attachment 1122871 [details]
Errors on ovirt engine
Comment 9 nicolas 2016-02-10 13:40 EST
Created attachment 1122872 [details]
Network traffic between engine and fake host
Comment 10 Piotr Kliczewski 2016-02-11 02:25:36 EST
Please make sure if you use secure connection that it is secured on both ends. Please provide fake vdsm logs.
Comment 11 nicolas 2016-02-11 13:22:11 EST
I have tried both with secure connection enabled at both ends and disabled on both ends (the last network traffic log was captured when I disabled it so I could read the commands being sent from the engine to the fake host).

I cannot provide any logs since the log directory is empty, only entries I see are (/var/log/vdsmfake/app.log):

2016-02-11 19:16:11,523 [main] INFO  - Application initialized.
2016-02-11 19:16:11,530 [main] DEBUG - Opening a Stomp server 192.168.1.X:54322

And I see directories created under /var/log/vdsmfake/xml but empty all of them.

It looks like this version of vdsmfake (latest from git://gerrit.ovirt.org/ovirt-vdsmfake.git) is not accepting commands that version 3.6.x of engine is sending?
Comment 12 Piotr Kliczewski 2016-02-12 02:31:19 EST
3.6 support is work in progress [1]. Please try with 3.5 engine or apply patches which are currently in progress.

[1] https://gerrit.ovirt.org/#/c/43811/
Comment 13 nicolas 2016-02-12 16:02:08 EST
We run 3.6 so I had the patches way.

I applied the mentioned patches (git fetch git://gerrit.ovirt.org/ovirt-vdsmfake refs/changes/11/43811/18 && git checkout FETCH_HEAD), however, it doesn't make any difference.

Still I had to modify my pom.xml to set version of org.ovirt.vdsm-jsonrpc-java to 1.1.8 as the one provided in the repo (1.1.2-SNAPSHOT) is no longer available. I wonder if this makes a difference.

Only to make sure this is not something network related I cloned the repo in the same engine machine and changed the host to (replaced the /etc/hosts entry as well), but still got the "HTTP/1.1 400 Bad Request" messages.

If this work for you and if you prefer, we can close this report and I'll investigate these days calmly to see if the issue isn't somewhere on my side.
Comment 14 Piotr Kliczewski 2016-02-15 02:36:00 EST
Eldad, please give guidelines how to make it working with 3.6
Comment 15 nicolas 2016-02-17 13:31:33 EST
I could finally make it work with some patches provided by Roman Mohr. As stated on the users list:

git clone git://gerrit.ovirt.org/ovirt-vdsmfake
cd ovirt-vdsmfake
git fetch git://gerrit.ovirt.org/ovirt-vdsmfake refs/changes/70/53570/2 && git checkout FETCH_HEAD

And followed the instructions detailed here [1].

Thanks for the support.

 [1]: https://gerrit.ovirt.org/#/c/53570/2/README
Comment 16 Oved Ourfali 2016-02-18 08:15:09 EST
Fake vdsm isn't tracked via bugzilla.
Please continue discussing offline / irc / emails.

Note You need to log in before you can comment on or make changes to this bug.