Bug 987439 - watchdog device created without model and action
Summary: watchdog device created without model and action
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-webadmin-portal
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: ---
: 3.3.0
Assignee: Martin Polednik
QA Contact: Lukas Svaty
URL:
Whiteboard: virt
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-23 11:37 UTC by Lukas Svaty
Modified: 2013-10-02 09:45 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-10-02 09:45:00 UTC
oVirt Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
engine.log (5.82 KB, text/x-log)
2013-07-23 11:37 UTC, Lukas Svaty
no flags Details
libvirtd.log (12.27 MB, text/x-log)
2013-07-26 14:40 UTC, Lukas Svaty
no flags Details
libvirtd log of hypervisor (12.37 KB, text/x-log)
2013-08-15 15:06 UTC, Lukas Svaty
no flags Details
engine log when VM started of hypervisor (10.90 KB, text/x-log)
2013-08-15 15:07 UTC, Lukas Svaty
no flags Details
vdsm.log of hypervisor (43.93 KB, text/x-log)
2013-08-15 15:07 UTC, Lukas Svaty
no flags Details
engine log when Vm tried to start on RHEL (10.89 KB, text/x-log)
2013-08-15 15:08 UTC, Lukas Svaty
no flags Details
vdsm.log from rhel (35.83 KB, text/x-log)
2013-08-15 15:08 UTC, Lukas Svaty
no flags Details

Description Lukas Svaty 2013-07-23 11:37:44 UTC
Created attachment 777286 [details]
engine.log

Description of problem:
Launching Vm fails.

Version-Release number of selected component (if applicable):
is6

How reproducible:
100%

Steps to Reproduce:
1. Add Cluster with ovirt service enabled and host
2. Add new Vm
3. Launch it

Actual results:
Vm failed to run.

Expected results:
Vm should be runnable

Additional info:

Comment 1 Michal Skrivanek 2013-07-24 13:27:50 UTC
what VM, on what host, vdsm/libvirt logs? man, come on...

anyway, error coming from libvirt - 'ebtables tool is missing', related to nwfilter -> moving to network

Comment 2 Lukas Svaty 2013-07-24 18:47:34 UTC
could not reproduce it in 3.3 compatiblity cluster today... happend again in 3.2 
adding vdsm and libvirt logs 

host: RHEV Hypervisor - 6.4 - 20130528.0.el6_4

root@localhost : rpm -qa | grep vdsm
vdsm-xmlrpc-4.10.2-22.0.el6ev.noarch
vdsm-cli-4.10.2-22.0.el6ev.noarch
vdsm-4.10.2-22.0.el6ev.x86_64
vdsm-hook-vhostmd-4.10.2-22.0.el6ev.noarch
vdsm-reg-4.10.2-22.0.el6ev.noarch
vdsm-python-4.10.2-22.0.el6ev.x86_64

root@localhost : rpm -qa | grep libvirt
libvirt-lock-sanlock-0.10.2-18.el6_4.5.x86_64
libvirt-client-0.10.2-18.el6_4.5.x86_64
libvirt-python-0.10.2-18.el6_4.5.x86_64
libvirt-cim-0.6.1-4.el6.x86_64
libvirt-0.10.2-18.el6_4.5.x86_64

Comment 5 Dan Kenigsberg 2013-07-26 14:20:03 UTC
Given "VM vm is down. Exit message: internal error cannot create rule since ebtables tool is missing..", would you see if `ebtables` is installed on your host and executable by libvirt?

Comment 6 Lukas Svaty 2013-07-26 14:40:02 UTC
Created attachment 778786 [details]
libvirtd.log

adding libvirtd.log

Comment 7 Dan Kenigsberg 2013-07-26 14:55:28 UTC
What's your `virsh -r nwfilter-dumpxml vdsm-no-mac-spoofing`?

Does it happen on any other host?

The ebtables failure gives me no clue, maybe Laine can chip in:

2013-07-23 12:57:57.331+0000: 12965: debug : virCommandRunAsync:2227 : Command result 0, with PID 19313
2013-07-23 12:57:57.345+0000: 12965: debug : virCommandRun:2025 : Result exit status 255, stdout: '' stderr: 'Illegal target name 'libvirt-J-vnet0'.
Illegal target name 'libvirt-P-vnet0'.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.

Can you reproduce this with `virsh` directly with a domxml similar to

<domain type="kvm">
	<name>watchdog_vm0</name>
	<uuid>0c059292-d85d-4b83-8cb0-75fc90271ded</uuid>
	<memory>1048576</memory>
	<currentMemory>1048576</currentMemory>
	<vcpu>1</vcpu>
	<devices>
		<channel type="unix">
			<target name="com.redhat.rhevm.vdsm" type="virtio"/>
			<source mode="bind" path="/var/lib/libvirt/qemu/channels/watchdog_vm0.com.redhat.rhevm.vdsm"/>
		</channel>
		<channel type="unix">
			<target name="org.qemu.guest_agent.0" type="virtio"/>
			<source mode="bind" path="/var/lib/libvirt/qemu/channels/watchdog_vm0.org.qemu.guest_agent.0"/>
		</channel>
		<input bus="ps2" type="mouse"/>
		<channel type="spicevmc">
			<target name="com.redhat.spice.0" type="virtio"/>
		</channel>
		<graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
			<channel mode="secure" name="main"/>
			<channel mode="secure" name="inputs"/>
			<channel mode="secure" name="cursor"/>
			<channel mode="secure" name="playback"/>
			<channel mode="secure" name="record"/>
			<channel mode="secure" name="display"/>
			<channel mode="secure" name="usbredir"/>
			<channel mode="secure" name="smartcard"/>
		</graphics>
		<sound model="ich6"/>
		<controller model="virtio-scsi" type="scsi"/>
		<video>
			<model heads="1" type="qxl" vram="65536"/>
		</video>
		<interface type="bridge">
			<mac address="00:1a:4a:22:3f:dc"/>
			<model type="virtio"/>
			<source bridge="rhevm"/>
			<filterref filter="vdsm-no-mac-spoofing"/>
			<link state="up"/>
			<boot order="1"/>
		</interface>
		<memballoon model="virtio"/>
		<disk device="cdrom" snapshot="no" type="file">
			<source file="" startupPolicy="optional"/>
			<target bus="ide" dev="hdc"/>
			<readonly/>
			<serial></serial>
		</disk>
	</devices>
	<os>
		<type arch="x86_64" machine="rhel6.4.0">hvm</type>
		<smbios mode="sysinfo"/>
	</os>
	<sysinfo type="smbios">
		<system>
			<entry name="manufacturer">Red Hat</entry>
			<entry name="product">RHEV Hypervisor</entry>
			<entry name="version">6.4-20130528.0.el6_4</entry>
			<entry name="serial">4C4C4544-0034-5310-8052-B1C04F4A354A</entry>
			<entry name="uuid">0c059292-d85d-4b83-8cb0-75fc90271ded</entry>
		</system>
	</sysinfo>
	<clock adjustment="0" offset="variable">
		<timer name="rtc" tickpolicy="catchup"/>
	</clock>
	<features>
		<acpi/>
	</features>
	<cpu match="exact">
		<model>Conroe</model>
		<topology cores="1" sockets="1" threads="1"/>
	</cpu>
</domain>

Comment 8 Lukas Svaty 2013-07-26 15:41:07 UTC
now running is7

at the moment couldn't reproduce this

[root@localhost ~]# virsh -r nwfilter-dumpxml vdsm-no-mac-spoofing
<filter name='vdsm-no-mac-spoofing' chain='root'>
  <uuid>2857b9e3-0f05-c413-c30d-ba143697d71d</uuid>
  <filterref filter='no-mac-spoofing'/>
  <filterref filter='no-arp-mac-spoofing'/>
</filter>

Comment 9 Dan Kenigsberg 2013-07-26 19:23:53 UTC
can you find and attach a log-rotated libvirtd.log with the string "ebtables tool is missing"?

Comment 10 Lukas Svaty 2013-08-06 10:36:29 UTC
went through log-rotated libvirtd.log and "ebtables tool is missing" was not in found

Comment 11 Dan Kenigsberg 2013-08-06 20:01:32 UTC
Please reopen if it ever reproduces (in a way other than deleting /usr/sbin/ebtables from the file system).

Comment 12 Lukas Svaty 2013-08-15 15:06:21 UTC
Created attachment 786967 [details]
libvirtd log of hypervisor

Comment 13 Lukas Svaty 2013-08-15 15:07:11 UTC
Created attachment 786968 [details]
engine log when VM started of hypervisor

Comment 14 Lukas Svaty 2013-08-15 15:07:43 UTC
Created attachment 786969 [details]
vdsm.log of hypervisor

Comment 15 Lukas Svaty 2013-08-15 15:08:18 UTC
Created attachment 786971 [details]
engine log when Vm tried to start on RHEL

Comment 16 Lukas Svaty 2013-08-15 15:08:44 UTC
Created attachment 786972 [details]
vdsm.log from rhel

Comment 17 Lukas Svaty 2013-08-15 15:14:19 UTC
Reproduced this bug on is10 in 3.2 cluster

When set option "Start running on specific host" for VM

Vm actually tried to start running on the other host

Set Start Running on specific host: 10.34.62.203 - RHEL
Vm Started running on RHEV-H host (10.34.62.204) and was shut down in less than 1 second

logs:
engine.log of this action: engine-rhevh.log
vdsm.log of this action: vdsm-rhevh.log
libvirtd.log of this action: libvirtd-rhevh.log

When I set Start Running on specific host on 10.34.62.204 - RHEVH
Vm tried runnning on 10.34.62.203 my RHEL host

logs
engine.log of this action: engine-rhel.log
vdsm.log of this action: vdsm-rhel.log
libvirtd.log of this action: nothing appeared in libvirtd.log

This may be two bugs one that its starting on another host and that it is not actually starting the VM

This bug is not that easy to reproduce so if somebody want to try for himself I can provide my setup while its still reproducable.

Comment 18 Dan Kenigsberg 2013-08-23 12:13:57 UTC
Why do you consider this reproduction as the same bug? I see a completely different error, unrelated to ebtables.


Thread-963::DEBUG::2013-08-15 14:57:26,105::BindingXMLRPC::920::vds::(wrapper) return vmCreate with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.4.0', 'vmId': '3ba9eb8e-bb15-4219-800a-c80c3badb6e5', 'pid': '0', 'memGuaranteedSize': 1024, 'timeOffset': '0', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'SandyBridge', ...., 'clientIp': '', 'nicModel': 'rtl8139,pv', 'smartcardEnable': 'false', 'kvmEnable': 'true', 'transparentHugePages': 'true', 'devices': [{'device': 'qxl', 'specParams': {'vram': '65536', 'heads': '1'}, 'type': 'video', 'deviceId': '19271706-bfe1-482c-8a7d-4d6b15c9f4b3'}, {'index': '2', 'iface': 'ide', 'specParams': {'path': ''}, 'readonly': 'true', 'deviceId': 'a33c4d4e-51db-490f-a5c3-ec39214d977a', 'path': '', 'device': 'cdrom', 'shared': 'false', 'type': 'disk'}, {'index': 0, 'iface': 'virtio', 'format': 'raw', 'type': 'disk', 'volumeID': '8aa7bf39-4be2-44e5-806e-85557c575c75', 'imageID': '2cb74bce-7583-4240-8e16-e71dcfa0b4fa', 'specParams': {}, 'readonly': 'false', 'domainID': '11f24381-303a-4b86-97d9-0aa1d53e7963', 'deviceId': '2cb74bce-7583-4240-8e16-e71dcfa0b4fa', 'poolID': '18bba25b-3a1e-47d8-be04-eb54c59090b8', 'device': 'disk', 'shared': 'false', 'propagateErrors': 'off', 'optional': 'false'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:1a:4a:22:3f:dc', 'linkActive': 'true', 'network': 'rhevm', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': '9964bfed-f960-480f-a80f-aec13dcf26ae', 'device': 'bridge', 'type': 'interface'}, {'device': 'memballoon', 'specParams': {'model': 'virtio'}, 'type': 'balloon', 'deviceId': '53f3bb96-5fa5-405b-872b-f0cf31dfb0ea'}, {'device': 'watchdog', 'specParams': {'action': 'poweroff', 'model': 'ib700'}, 'type': 'watchdog', 'deviceId': '00000000-0000-0000-0000-000000000000'}], 'smp': '2', 'vmType': 'kvm', 'memSize': 8182, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', 'smpCoresPerSocket': '1', 'vmName': 'vm', 'display': 'qxl', 'nice': '0'}}
Thread-964::INFO::2013-08-15 14:57:26,105::libvirtvm::1453::vm.Vm::(_run) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::VM wrapper has started

Thread-964::WARNING::2013-08-15 14:57:26,106::vm::457::vm.Vm::(getConfDevices) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::Unknown type found, device: '{'device': 'watchdog', 'specParams': {'action': 'poweroff', 'model': 'ib700'}, 'type': 'watchdog', 'deviceId': '00000000-0000-0000-0000-000000000000'}' found


...
		<watchdog type="watchdog"/>
	</devices>
...

Thread-964::ERROR::2013-08-15 14:57:26,172::vm::704::vm.Vm::(_startUnderlyingVm) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 664, in _startUnderlyingVm
  File "/usr/share/vdsm/libvirtvm.py", line 1535, in _run
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 104, in wrapper
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2645, in createXML
libvirtError: internal error watchdog must contain model name
Thread-964::DEBUG::2013-08-15 14:57:26,175::vm::1092::vm.Vm::(setDownStatus) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::Changed state to Down: internal error watchdog must contain model name

Comment 19 Michal Skrivanek 2013-09-05 12:50:58 UTC
Martin, pls check also to regard with your latest patch in this area

Comment 20 Michal Skrivanek 2013-09-11 11:15:44 UTC
- specify versions
- do not reopen bugs on different issue. It only causes chaos and delays

thanks

Comment 21 Lukas Svaty 2013-10-02 09:14:03 UTC
tried reproducing this in version is17, could not reproduce adding watchdog without model or/and action via admin portal/rest api

libvirt-0.10.2-27.el6.x86_64 (host)

I suggest setting it to closed-current release

Comment 22 Michal Skrivanek 2013-10-02 09:45:00 UTC
thanks for checking it!


Note You need to log in before you can comment on or make changes to this bug.