Bug 1027692

Summary: Unable to start VM - EncryptHostCommunication=false conflicts with secure spice
Product: Red Hat Enterprise Virtualization Manager Reporter: Jiri Belka <jbelka>
Component: ovirt-engineAssignee: Alon Bar-Lev <alonbl>
Status: CLOSED WONTFIX QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: acathrow, alonbl, bazulay, emesika, iheim, jbelka, lpeer, michal.skrivanek, pstehlik, Rhev-m-bugs, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-11 12:16:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine.log, vdsm.log, libvirtd.log
none
engine.log, ovirt-20131113120313-10.34.62.205-13148100.log none

Description Jiri Belka 2013-11-07 09:57:35 UTC
Created attachment 820973 [details]
engine.log, vdsm.log, libvirtd.log

Description of problem:
If you define EncryptHostCommunication=false and _THEN_ you add new host into setup, you will see following lines in logs:

2013-11-07 09:09:57.317+0000: 8549: error : qemuBuildGraphicsCommandLine:4899 : unsupported configuration: spice secure channels set in XML configuration, but TLS is disabled in qemu.conf

I discovered this while having problem to start any VM on such host. Not sure if this error is starting point for all the issue but it does exist as error...

* From a host where it fails/error exists in libvirtd.log:

-%-
# egrep -Ri "ssl|tls|secure" /etc/libvirt* | grep -v :#
/etc/libvirt/qemu.conf:spice_tls=0
/etc/libvirt/libvirtd.conf:listen_tls=0
# egrep -Ri "ssl|tls|secure" /etc/vdsm* | grep -v :#
/etc/vdsm/vdsm.conf:ssl = false
-%-

* From a host where I can start VMs:

-%-
# egrep -Ri "ssl|tls|secure" /etc/libvirt* | grep -v :#
/etc/libvirt/qemu.conf:spice_tls=1
/etc/libvirt/qemu.conf:spice_tls_x509_cert_dir="/etc/pki/vdsm/libvirt-spice"
# egrep -Ri "ssl|tls|secure" /etc/vdsm* | grep -v :#
/etc/vdsm/vdsm.conf:ssl = false
-%-

As you can see on problematic host 'listen_tls=0' exists, while on working host it is in default (listen_tls enabled).

So basically I would say that EncryptHostCommunication=false is too pro-active. It does not only disable ssl between RHEVM and vdsm but also ssl for libvirt. The original request from a customer (BZ1003117) was to disable secure communication between RHEVM and hosts.

> 4. Why does the customer need this? (List the business requirements here)
> Customer needs to disable ssl communication between RHEV-M and the hosts for
> his internal tests/implementation every now and then.

Thus engine is inconsistent. It disables too much but it sends VM data containing secure spice...

-%-
2013-11-07 10:36:12,911 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-50) [6f294710] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=1024,kvmEnable=true,smp=1,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=en-us,memGuaranteedSize=1024,pitReinjection=false,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=0,transparentHugePages=true,vmId=ad4556db-85bd-4498-97fd-bb734fd50060,devices=[Ljava.util.HashMap;@52382c55,acpiEnable=true,vmName=test-rh5-x86,cpuType=Conroe,custom={}
-%-

...which is following xml part on vdsm side:

-%-
<domain type="kvm">
        <name>test-rh5-x86</name>
        <uuid>ad4556db-85bd-4498-97fd-bb734fd50060</uuid>
        <memory>1048576</memory>
        <currentMemory>1048576</currentMemory>
        <vcpu>1</vcpu>
        <memtune>
                <min_guarantee>1048576</min_guarantee>
        </memtune>
        <devices>
                <channel type="unix">
                        <target name="com.redhat.rhevm.vdsm" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/ad4556db-85bd-4498-97fd-bb734fd50060.com.redhat.rhevm.vdsm"/>
                </channel>
                <channel type="unix">
                        <target name="org.qemu.guest_agent.0" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/ad4556db-85bd-4498-97fd-bb734fd50060.org.qemu.guest_agent.0"/>
                </channel>
                <input bus="ps2" type="mouse"/>
                <channel type="spicevmc">
                        <target name="com.redhat.spice.0" type="virtio"/>
                </channel>
                <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
                        <channel mode="secure" name="main"/>
                        <channel mode="secure" name="inputs"/>
                        <channel mode="secure" name="cursor"/>
                        <channel mode="secure" name="playback"/>
                        <channel mode="secure" name="record"/>
                        <channel mode="secure" name="display"/>
                        <channel mode="secure" name="usbredir"/>
                        <channel mode="secure" name="smartcard"/>
                </graphics>
...snip...
-%-


Version-Release number of selected component (if applicable):
is21
qemu-kvm-rhev-0.12.1.2-2.414.el6.x86_64
libvirt-0.10.2-29.el6.x86_64
vdsm-4.13.0-0.5.beta1.el6ev.x86_64
redhat-release-server-6Server-6.5.0.1.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. engine-config -s EncryptHostCommunication=false
2. restart rhevm
3. add host

Actual results:
unable to start vm

Expected results:
plain text rhevm<->host communication should not impact secure spice and vm should start up

Additional info:
please clarify if this issue impacts whole VM startup

Comment 1 Jiri Belka 2013-11-07 10:36:44 UTC
I cannot find easy way for a workaround. Changing values in conf file is not probably enough, the host stays in Initializing state and I cannot do anything with that...

Comment 2 Jiri Belka 2013-11-07 10:39:17 UTC
So after struggle I have it in Non-Operational but I can't do anything with that.

Comment 3 Andrew Cathrow 2013-11-12 13:48:13 UTC
Itamar / Barak - thoughts on this - did we go too far - should this setting, as RFEd, only be for host communication - hence an Infra bug?

Comment 4 Andrew Cathrow 2013-11-12 15:14:03 UTC
I'm inclined to say that the fix should be infra and to only change the engine->vdsm config.

Comment 5 Barak 2013-11-12 19:03:57 UTC
(In reply to Andrew Cathrow from comment #4)
> I'm inclined to say that the fix should be infra and to only change the
> engine->vdsm config.

This is pure virt, the execution of a VM is entirely virt.

Comment 6 Itamar Heim 2013-11-13 10:10:18 UTC
there was a reason this key wasn't public - it was meant to allow developers to work without ssl/certificates before we had the current dev environment.
i.e., this key doesn't disable ssl only, it bypsases the certificate generation/handling, so spice can't use ssl as well.
read: this disable all host encrypted communication including spice, not just engine-host.

so today, it should only changed to false together with SSLEnabled (which also deserves a much better name...).

to really allow changing just one of them, then yes, infra need to differentiate between EncryptHostCommunication and certificate generation.
the virt aspect should already be covered by setting SSLEnabled=false.

jiri - did you try SSLEnabled=false?

Comment 7 Jiri Belka 2013-11-13 11:17:42 UTC
Created attachment 823364 [details]
engine.log, ovirt-20131113120313-10.34.62.205-13148100.log

No I did not try it. How would I know that? I'm doing what a customer would do, one who requested EncryptHostCommunication being configurable via engine-config.

Result when adding host while having both mentioned knobs set to false on RHEVM (installation failed because "FAILED: conflicting vdsm and libvirt-qemu tls configuration.", host-deploy issue...):

-%-
2013-11-13 11:03:13 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:441 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout:
supervdsm start[  OK  ]
vdsm: Running run_init_hooks
vdsm: Running gencerts
vdsm: Running reconfigure_sanlock
vdsm: Running reconfigure_libvirt
checking certs..
libvirt is already configured for vdsm
vdsm: Running prepare_transient_repository
vdsm: Running syslog_available
vdsm: Running nwfilter
vdsm: Running dummybr
vdsm: Running load_needed_modules
vdsm: Running tune_system
vdsm: Running mkdirs
vdsm: Running test_space
vdsm: Running test_lo
vdsm: Running test_conflicting_conf
FAILED: conflicting vdsm and libvirt-qemu tls configuration.
vdsm.conf with ssl=False requires the following changed:
libvirtd.conf: listen_tcp=1, auth_tcp="none",
qemu.conf: spice_tls=0.
vdsm: failed to execute test_conflicting_conf, error code 1
vdsm start[FAILED]

2013-11-13 11:03:13 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:446 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr:
initctl: Job is already running: libvirtd
Traceback (most recent call last):
  File "/usr/bin/vdsm-tool", line 143, in <module>
    sys.exit(main())
  File "/usr/bin/vdsm-tool", line 140, in main
    return tool_command[cmd]["command"](*args[1:])
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/libvirt_configure.py", line 65, in test_conflict_configurations
  File "/usr/lib64/python2.6/site-packages/vdsm/tool/libvirt_configure.py", line 46, in exec_libvirt_configure
RuntimeError: Failed to configure libvirt

2013-11-13 11:03:13 DEBUG otopi.context context._executeMethod:137 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-jq8UsKR78D/pythonlib/otopi/context.py", line 127, in _executeMethod
    method['method']()
  File "/tmp/ovirt-jq8UsKR78D/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", line 217, in _start
    self.services.state('vdsmd', True)
  File "/tmp/ovirt-jq8UsKR78D/otopi-plugins/otopi/services/rhel.py", line 188, in state
    'start' if state else 'stop'
  File "/tmp/ovirt-jq8UsKR78D/otopi-plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand
    raiseOnError=raiseOnError
  File "/tmp/ovirt-jq8UsKR78D/pythonlib/otopi/plugin.py", line 451, in execute
    command=args[0],
RuntimeError: Command '/sbin/service' failed to execute
2013-11-13 11:03:13 ERROR otopi.context context._executeMethod:146 Failed to execute stage 'Closing up': Command '/sbin/service' failed to execute
-%-

Comment 8 Jiri Belka 2013-11-13 11:40:12 UTC
I got it working with following steps:

* install failed
* remove host
* modify below options
* add host

-%-
# egrep "ssl|listen_tcp|auth_tcp|spice_tls[ \t]+" /etc/libvirt/libvirtd.conf /etc/libvirt/qemu.conf /etc/vdsm/vdsm.conf | grep -v :#
/etc/libvirt/libvirtd.conf:listen_tcp = 1
/etc/libvirt/libvirtd.conf:auth_tcp = "none"
/etc/libvirt/qemu.conf:spice_tls = 0
/etc/vdsm/vdsm.conf:ssl = false
-%-

VM has only plain-text spice:

-%-
# lsof -nPc qemu | grep LISTEN
qemu-kvm 15475 qemu   18u  IPv4             183586         0t0     TCP *:5900 (LISTEN
-%-

SPICE console was opened to plain-text spice socket:

-%-
# tcpdump -i eth0 -n -s 1023 -A -c 10 host 10.34.62.205
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 1023 bytes
12:32:17.745623 IP 10.34.131.48.56301 > 10.34.62.205.vnc-server: Flags [S], seq 1922054203, win 14600, options [mss 1460,sackOK,TS val 239114907 ecr 0,nop,wsca
le 7], length 0
E..<..@.@...
".0
">.....r.8;......9............
.@..........
-%-

But to spice client even secure spice port is reported... Why if it does not exist?

-%-
$ grep -i port .spicec/spice-xpi.log
2013-11-13 12:34:33,359 DEBUG nsPluginInstance::SetPort: 5900
2013-11-13 12:34:33,361 DEBUG nsPluginInstance::SetUsbListenPort: 0
2013-11-13 12:34:33,362 DEBUG nsPluginInstance::SetSecurePort: 65535
-%-

Comment 9 Alon Bar-Lev 2013-11-19 07:05:34 UTC
See discussion in bug#1026300, the request at bug#1003117 is totally invalid for production, including this one.

Comment 10 Alon Bar-Lev 2013-12-08 18:50:30 UTC
Hello,

I want to clean this as WONTFIX, if anyone has something to say, please speak now.

Thanks,