Bug 1027692 - Unable to start VM - EncryptHostCommunication=false conflicts with secure spice
Unable to start VM - EncryptHostCommunication=false conflicts with secure spice
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
Unspecified Unspecified
unspecified Severity urgent
: ---
: 3.4.0
Assigned To: Alon Bar-Lev
: Triaged
Depends On:
  Show dependency treegraph
Reported: 2013-11-07 04:57 EST by Jiri Belka
Modified: 2016-02-10 14:32 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-12-11 07:16:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
engine.log, vdsm.log, libvirtd.log (193.25 KB, application/x-gzip)
2013-11-07 04:57 EST, Jiri Belka
no flags Details
engine.log, ovirt-20131113120313- (576.60 KB, application/x-gzip)
2013-11-13 06:17 EST, Jiri Belka
no flags Details

  None (edit)
Description Jiri Belka 2013-11-07 04:57:35 EST
Created attachment 820973 [details]
engine.log, vdsm.log, libvirtd.log

Description of problem:
If you define EncryptHostCommunication=false and _THEN_ you add new host into setup, you will see following lines in logs:

2013-11-07 09:09:57.317+0000: 8549: error : qemuBuildGraphicsCommandLine:4899 : unsupported configuration: spice secure channels set in XML configuration, but TLS is disabled in qemu.conf

I discovered this while having problem to start any VM on such host. Not sure if this error is starting point for all the issue but it does exist as error...

* From a host where it fails/error exists in libvirtd.log:

# egrep -Ri "ssl|tls|secure" /etc/libvirt* | grep -v :#
# egrep -Ri "ssl|tls|secure" /etc/vdsm* | grep -v :#
/etc/vdsm/vdsm.conf:ssl = false

* From a host where I can start VMs:

# egrep -Ri "ssl|tls|secure" /etc/libvirt* | grep -v :#
# egrep -Ri "ssl|tls|secure" /etc/vdsm* | grep -v :#
/etc/vdsm/vdsm.conf:ssl = false

As you can see on problematic host 'listen_tls=0' exists, while on working host it is in default (listen_tls enabled).

So basically I would say that EncryptHostCommunication=false is too pro-active. It does not only disable ssl between RHEVM and vdsm but also ssl for libvirt. The original request from a customer (BZ1003117) was to disable secure communication between RHEVM and hosts.

> 4. Why does the customer need this? (List the business requirements here)
> Customer needs to disable ssl communication between RHEV-M and the hosts for
> his internal tests/implementation every now and then.

Thus engine is inconsistent. It disables too much but it sends VM data containing secure spice...

2013-11-07 10:36:12,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) [6f294710] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=1024,kvmEnable=true,smp=1,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=en-us,memGuaranteedSize=1024,pitReinjection=false,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=0,transparentHugePages=true,vmId=ad4556db-85bd-4498-97fd-bb734fd50060,devices=[Ljava.util.HashMap;@52382c55,acpiEnable=true,vmName=test-rh5-x86,cpuType=Conroe,custom={}

...which is following xml part on vdsm side:

<domain type="kvm">
                <channel type="unix">
                        <target name="com.redhat.rhevm.vdsm" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/ad4556db-85bd-4498-97fd-bb734fd50060.com.redhat.rhevm.vdsm"/>
                <channel type="unix">
                        <target name="org.qemu.guest_agent.0" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/ad4556db-85bd-4498-97fd-bb734fd50060.org.qemu.guest_agent.0"/>
                <input bus="ps2" type="mouse"/>
                <channel type="spicevmc">
                        <target name="com.redhat.spice.0" type="virtio"/>
                <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
                        <channel mode="secure" name="main"/>
                        <channel mode="secure" name="inputs"/>
                        <channel mode="secure" name="cursor"/>
                        <channel mode="secure" name="playback"/>
                        <channel mode="secure" name="record"/>
                        <channel mode="secure" name="display"/>
                        <channel mode="secure" name="usbredir"/>
                        <channel mode="secure" name="smartcard"/>

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. engine-config -s EncryptHostCommunication=false
2. restart rhevm
3. add host

Actual results:
unable to start vm

Expected results:
plain text rhevm<->host communication should not impact secure spice and vm should start up

Additional info:
please clarify if this issue impacts whole VM startup
Comment 1 Jiri Belka 2013-11-07 05:36:44 EST
I cannot find easy way for a workaround. Changing values in conf file is not probably enough, the host stays in Initializing state and I cannot do anything with that...
Comment 2 Jiri Belka 2013-11-07 05:39:17 EST
So after struggle I have it in Non-Operational but I can't do anything with that.
Comment 3 Andrew Cathrow 2013-11-12 08:48:13 EST
Itamar / Barak - thoughts on this - did we go too far - should this setting, as RFEd, only be for host communication - hence an Infra bug?
Comment 4 Andrew Cathrow 2013-11-12 10:14:03 EST
I'm inclined to say that the fix should be infra and to only change the engine->vdsm config.
Comment 5 Barak 2013-11-12 14:03:57 EST
(In reply to Andrew Cathrow from comment #4)
> I'm inclined to say that the fix should be infra and to only change the
> engine->vdsm config.

This is pure virt, the execution of a VM is entirely virt.
Comment 6 Itamar Heim 2013-11-13 05:10:18 EST
there was a reason this key wasn't public - it was meant to allow developers to work without ssl/certificates before we had the current dev environment.
i.e., this key doesn't disable ssl only, it bypsases the certificate generation/handling, so spice can't use ssl as well.
read: this disable all host encrypted communication including spice, not just engine-host.

so today, it should only changed to false together with SSLEnabled (which also deserves a much better name...).

to really allow changing just one of them, then yes, infra need to differentiate between EncryptHostCommunication and certificate generation.
the virt aspect should already be covered by setting SSLEnabled=false.

jiri - did you try SSLEnabled=false?
Comment 7 Jiri Belka 2013-11-13 06:17:42 EST
Created attachment 823364 [details]
engine.log, ovirt-20131113120313-

No I did not try it. How would I know that? I'm doing what a customer would do, one who requested EncryptHostCommunication being configurable via engine-config.

Result when adding host while having both mentioned knobs set to false on RHEVM (installation failed because "FAILED: conflicting vdsm and libvirt-qemu tls configuration.", host-deploy issue...):

2013-11-13 11:03:13 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:441 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout:
supervdsm start[  OK  ]
vdsm: Running run_init_hooks
vdsm: Running gencerts
vdsm: Running reconfigure_sanlock
vdsm: Running reconfigure_libvirt
checking certs..
libvirt is already configured for vdsm
vdsm: Running prepare_transient_repository
vdsm: Running syslog_available
vdsm: Running nwfilter
vdsm: Running dummybr
vdsm: Running load_needed_modules
vdsm: Running tune_system
vdsm: Running mkdirs
vdsm: Running test_space
vdsm: Running test_lo
vdsm: Running test_conflicting_conf
FAILED: conflicting vdsm and libvirt-qemu tls configuration.
vdsm.conf with ssl=False requires the following changed:
libvirtd.conf: listen_tcp=1, auth_tcp="none",
qemu.conf: spice_tls=0.
vdsm: failed to execute test_conflicting_conf, error code 1
vdsm start[FAILED]

2013-11-13 11:03:13 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:446 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr:
initctl: Job is already running: libvirtd
Traceback (most recent call last):
  File "/usr/bin/vdsm-tool", line 143, in <module>
  File "/usr/bin/vdsm-tool", line 140, in main
    return tool_command[cmd]["command"](*args[1:])
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/libvirt_configure.py", line 65, in test_conflict_configurations
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/libvirt_configure.py", line 46, in exec_libvirt_configure
RuntimeError: Failed to configure libvirt

2013-11-13 11:03:13 DEBUG otopi.context context._executeMethod:137 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-jq8UsKR78D/pythonlib/otopi/context.py", line 127, in _executeMethod
  File "/tmp/ovirt-jq8UsKR78D/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 217, in _start
    self.services.state('vdsmd', True)
  File "/tmp/ovirt-jq8UsKR78D/otopi-plugins/otopi/services/rhel.py", line 188, in state
    'start' if state else 'stop'
  File "/tmp/ovirt-jq8UsKR78D/otopi-plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand
  File "/tmp/ovirt-jq8UsKR78D/pythonlib/otopi/plugin.py", line 451, in execute
RuntimeError: Command '/sbin/service' failed to execute
2013-11-13 11:03:13 ERROR otopi.context context._executeMethod:146 Failed to execute stage 'Closing up': Command '/sbin/service' failed to execute
Comment 8 Jiri Belka 2013-11-13 06:40:12 EST
I got it working with following steps:

* install failed
* remove host
* modify below options
* add host

# egrep "ssl|listen_tcp|auth_tcp|spice_tls[ \t]+" /etc/libvirt/libvirtd.conf /etc/libvirt/qemu.conf /etc/vdsm/vdsm.conf | grep -v :#
/etc/libvirt/libvirtd.conf:listen_tcp = 1
/etc/libvirt/libvirtd.conf:auth_tcp = "none"
/etc/libvirt/qemu.conf:spice_tls = 0
/etc/vdsm/vdsm.conf:ssl = false

VM has only plain-text spice:

# lsof -nPc qemu | grep LISTEN
qemu-kvm 15475 qemu   18u  IPv4             183586         0t0     TCP *:5900 (LISTEN

SPICE console was opened to plain-text spice socket:

# tcpdump -i eth0 -n -s 1023 -A -c 10 host
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 1023 bytes
12:32:17.745623 IP > Flags [S], seq 1922054203, win 14600, options [mss 1460,sackOK,TS val 239114907 ecr 0,nop,wsca
le 7], length 0

But to spice client even secure spice port is reported... Why if it does not exist?

$ grep -i port .spicec/spice-xpi.log
2013-11-13 12:34:33,359 DEBUG nsPluginInstance::SetPort: 5900
2013-11-13 12:34:33,361 DEBUG nsPluginInstance::SetUsbListenPort: 0
2013-11-13 12:34:33,362 DEBUG nsPluginInstance::SetSecurePort: 65535
Comment 9 Alon Bar-Lev 2013-11-19 02:05:34 EST
See discussion in bug#1026300, the request at bug#1003117 is totally invalid for production, including this one.
Comment 10 Alon Bar-Lev 2013-12-08 13:50:30 EST

I want to clean this as WONTFIX, if anyone has something to say, please speak now.


Note You need to log in before you can comment on or make changes to this bug.