Bug 987439 - watchdog device created without model and action
watchdog device created without model and action
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-webadmin-portal (Show other bugs)
3.3.0
Unspecified Unspecified
medium Severity unspecified
: ---
: 3.3.0
Assigned To: Martin Polednik
Lukas Svaty
virt
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-23 07:37 EDT by Lukas Svaty
Modified: 2013-10-02 05:45 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-02 05:45:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine.log (5.82 KB, text/x-log)
2013-07-23 07:37 EDT, Lukas Svaty
no flags Details
libvirtd.log (12.27 MB, text/x-log)
2013-07-26 10:40 EDT, Lukas Svaty
no flags Details
libvirtd log of hypervisor (12.37 KB, text/x-log)
2013-08-15 11:06 EDT, Lukas Svaty
no flags Details
engine log when VM started of hypervisor (10.90 KB, text/x-log)
2013-08-15 11:07 EDT, Lukas Svaty
no flags Details
vdsm.log of hypervisor (43.93 KB, text/x-log)
2013-08-15 11:07 EDT, Lukas Svaty
no flags Details
engine log when Vm tried to start on RHEL (10.89 KB, text/x-log)
2013-08-15 11:08 EDT, Lukas Svaty
no flags Details
vdsm.log from rhel (35.83 KB, text/x-log)
2013-08-15 11:08 EDT, Lukas Svaty
no flags Details

  None (edit)
Description Lukas Svaty 2013-07-23 07:37:44 EDT
Created attachment 777286 [details]
engine.log

Description of problem:
Launching Vm fails.

Version-Release number of selected component (if applicable):
is6

How reproducible:
100%

Steps to Reproduce:
1. Add Cluster with ovirt service enabled and host
2. Add new Vm
3. Launch it

Actual results:
Vm failed to run.

Expected results:
Vm should be runnable

Additional info:
Comment 1 Michal Skrivanek 2013-07-24 09:27:50 EDT
what VM, on what host, vdsm/libvirt logs? man, come on...

anyway, error coming from libvirt - 'ebtables tool is missing', related to nwfilter -> moving to network
Comment 2 Lukas Svaty 2013-07-24 14:47:34 EDT
could not reproduce it in 3.3 compatiblity cluster today... happend again in 3.2 
adding vdsm and libvirt logs 

host: RHEV Hypervisor - 6.4 - 20130528.0.el6_4

root@localhost : rpm -qa | grep vdsm
vdsm-xmlrpc-4.10.2-22.0.el6ev.noarch
vdsm-cli-4.10.2-22.0.el6ev.noarch
vdsm-4.10.2-22.0.el6ev.x86_64
vdsm-hook-vhostmd-4.10.2-22.0.el6ev.noarch
vdsm-reg-4.10.2-22.0.el6ev.noarch
vdsm-python-4.10.2-22.0.el6ev.x86_64

root@localhost : rpm -qa | grep libvirt
libvirt-lock-sanlock-0.10.2-18.el6_4.5.x86_64
libvirt-client-0.10.2-18.el6_4.5.x86_64
libvirt-python-0.10.2-18.el6_4.5.x86_64
libvirt-cim-0.6.1-4.el6.x86_64
libvirt-0.10.2-18.el6_4.5.x86_64
Comment 5 Dan Kenigsberg 2013-07-26 10:20:03 EDT
Given "VM vm is down. Exit message: internal error cannot create rule since ebtables tool is missing..", would you see if `ebtables` is installed on your host and executable by libvirt?
Comment 6 Lukas Svaty 2013-07-26 10:40:02 EDT
Created attachment 778786 [details]
libvirtd.log

adding libvirtd.log
Comment 7 Dan Kenigsberg 2013-07-26 10:55:28 EDT
What's your `virsh -r nwfilter-dumpxml vdsm-no-mac-spoofing`?

Does it happen on any other host?

The ebtables failure gives me no clue, maybe Laine can chip in:

2013-07-23 12:57:57.331+0000: 12965: debug : virCommandRunAsync:2227 : Command result 0, with PID 19313
2013-07-23 12:57:57.345+0000: 12965: debug : virCommandRun:2025 : Result exit status 255, stdout: '' stderr: 'Illegal target name 'libvirt-J-vnet0'.
Illegal target name 'libvirt-P-vnet0'.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-J-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.
Chain 'libvirt-P-vnet0' doesn't exist.

Can you reproduce this with `virsh` directly with a domxml similar to

<domain type="kvm">
	<name>watchdog_vm0</name>
	<uuid>0c059292-d85d-4b83-8cb0-75fc90271ded</uuid>
	<memory>1048576</memory>
	<currentMemory>1048576</currentMemory>
	<vcpu>1</vcpu>
	<devices>
		<channel type="unix">
			<target name="com.redhat.rhevm.vdsm" type="virtio"/>
			<source mode="bind" path="/var/lib/libvirt/qemu/channels/watchdog_vm0.com.redhat.rhevm.vdsm"/>
		</channel>
		<channel type="unix">
			<target name="org.qemu.guest_agent.0" type="virtio"/>
			<source mode="bind" path="/var/lib/libvirt/qemu/channels/watchdog_vm0.org.qemu.guest_agent.0"/>
		</channel>
		<input bus="ps2" type="mouse"/>
		<channel type="spicevmc">
			<target name="com.redhat.spice.0" type="virtio"/>
		</channel>
		<graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
			<channel mode="secure" name="main"/>
			<channel mode="secure" name="inputs"/>
			<channel mode="secure" name="cursor"/>
			<channel mode="secure" name="playback"/>
			<channel mode="secure" name="record"/>
			<channel mode="secure" name="display"/>
			<channel mode="secure" name="usbredir"/>
			<channel mode="secure" name="smartcard"/>
		</graphics>
		<sound model="ich6"/>
		<controller model="virtio-scsi" type="scsi"/>
		<video>
			<model heads="1" type="qxl" vram="65536"/>
		</video>
		<interface type="bridge">
			<mac address="00:1a:4a:22:3f:dc"/>
			<model type="virtio"/>
			<source bridge="rhevm"/>
			<filterref filter="vdsm-no-mac-spoofing"/>
			<link state="up"/>
			<boot order="1"/>
		</interface>
		<memballoon model="virtio"/>
		<disk device="cdrom" snapshot="no" type="file">
			<source file="" startupPolicy="optional"/>
			<target bus="ide" dev="hdc"/>
			<readonly/>
			<serial></serial>
		</disk>
	</devices>
	<os>
		<type arch="x86_64" machine="rhel6.4.0">hvm</type>
		<smbios mode="sysinfo"/>
	</os>
	<sysinfo type="smbios">
		<system>
			<entry name="manufacturer">Red Hat</entry>
			<entry name="product">RHEV Hypervisor</entry>
			<entry name="version">6.4-20130528.0.el6_4</entry>
			<entry name="serial">4C4C4544-0034-5310-8052-B1C04F4A354A</entry>
			<entry name="uuid">0c059292-d85d-4b83-8cb0-75fc90271ded</entry>
		</system>
	</sysinfo>
	<clock adjustment="0" offset="variable">
		<timer name="rtc" tickpolicy="catchup"/>
	</clock>
	<features>
		<acpi/>
	</features>
	<cpu match="exact">
		<model>Conroe</model>
		<topology cores="1" sockets="1" threads="1"/>
	</cpu>
</domain>
Comment 8 Lukas Svaty 2013-07-26 11:41:07 EDT
now running is7

at the moment couldn't reproduce this

[root@localhost ~]# virsh -r nwfilter-dumpxml vdsm-no-mac-spoofing
<filter name='vdsm-no-mac-spoofing' chain='root'>
  <uuid>2857b9e3-0f05-c413-c30d-ba143697d71d</uuid>
  <filterref filter='no-mac-spoofing'/>
  <filterref filter='no-arp-mac-spoofing'/>
</filter>
Comment 9 Dan Kenigsberg 2013-07-26 15:23:53 EDT
can you find and attach a log-rotated libvirtd.log with the string "ebtables tool is missing"?
Comment 10 Lukas Svaty 2013-08-06 06:36:29 EDT
went through log-rotated libvirtd.log and "ebtables tool is missing" was not in found
Comment 11 Dan Kenigsberg 2013-08-06 16:01:32 EDT
Please reopen if it ever reproduces (in a way other than deleting /usr/sbin/ebtables from the file system).
Comment 12 Lukas Svaty 2013-08-15 11:06:21 EDT
Created attachment 786967 [details]
libvirtd log of hypervisor
Comment 13 Lukas Svaty 2013-08-15 11:07:11 EDT
Created attachment 786968 [details]
engine log when VM started of hypervisor
Comment 14 Lukas Svaty 2013-08-15 11:07:43 EDT
Created attachment 786969 [details]
vdsm.log of hypervisor
Comment 15 Lukas Svaty 2013-08-15 11:08:18 EDT
Created attachment 786971 [details]
engine log when Vm tried to start on RHEL
Comment 16 Lukas Svaty 2013-08-15 11:08:44 EDT
Created attachment 786972 [details]
vdsm.log from rhel
Comment 17 Lukas Svaty 2013-08-15 11:14:19 EDT
Reproduced this bug on is10 in 3.2 cluster

When set option "Start running on specific host" for VM

Vm actually tried to start running on the other host

Set Start Running on specific host: 10.34.62.203 - RHEL
Vm Started running on RHEV-H host (10.34.62.204) and was shut down in less than 1 second

logs:
engine.log of this action: engine-rhevh.log
vdsm.log of this action: vdsm-rhevh.log
libvirtd.log of this action: libvirtd-rhevh.log

When I set Start Running on specific host on 10.34.62.204 - RHEVH
Vm tried runnning on 10.34.62.203 my RHEL host

logs
engine.log of this action: engine-rhel.log
vdsm.log of this action: vdsm-rhel.log
libvirtd.log of this action: nothing appeared in libvirtd.log

This may be two bugs one that its starting on another host and that it is not actually starting the VM

This bug is not that easy to reproduce so if somebody want to try for himself I can provide my setup while its still reproducable.
Comment 18 Dan Kenigsberg 2013-08-23 08:13:57 EDT
Why do you consider this reproduction as the same bug? I see a completely different error, unrelated to ebtables.


Thread-963::DEBUG::2013-08-15 14:57:26,105::BindingXMLRPC::920::vds::(wrapper) return vmCreate with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.4.0', 'vmId': '3ba9eb8e-bb15-4219-800a-c80c3badb6e5', 'pid': '0', 'memGuaranteedSize': 1024, 'timeOffset': '0', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'SandyBridge', ...., 'clientIp': '', 'nicModel': 'rtl8139,pv', 'smartcardEnable': 'false', 'kvmEnable': 'true', 'transparentHugePages': 'true', 'devices': [{'device': 'qxl', 'specParams': {'vram': '65536', 'heads': '1'}, 'type': 'video', 'deviceId': '19271706-bfe1-482c-8a7d-4d6b15c9f4b3'}, {'index': '2', 'iface': 'ide', 'specParams': {'path': ''}, 'readonly': 'true', 'deviceId': 'a33c4d4e-51db-490f-a5c3-ec39214d977a', 'path': '', 'device': 'cdrom', 'shared': 'false', 'type': 'disk'}, {'index': 0, 'iface': 'virtio', 'format': 'raw', 'type': 'disk', 'volumeID': '8aa7bf39-4be2-44e5-806e-85557c575c75', 'imageID': '2cb74bce-7583-4240-8e16-e71dcfa0b4fa', 'specParams': {}, 'readonly': 'false', 'domainID': '11f24381-303a-4b86-97d9-0aa1d53e7963', 'deviceId': '2cb74bce-7583-4240-8e16-e71dcfa0b4fa', 'poolID': '18bba25b-3a1e-47d8-be04-eb54c59090b8', 'device': 'disk', 'shared': 'false', 'propagateErrors': 'off', 'optional': 'false'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:1a:4a:22:3f:dc', 'linkActive': 'true', 'network': 'rhevm', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': '9964bfed-f960-480f-a80f-aec13dcf26ae', 'device': 'bridge', 'type': 'interface'}, {'device': 'memballoon', 'specParams': {'model': 'virtio'}, 'type': 'balloon', 'deviceId': '53f3bb96-5fa5-405b-872b-f0cf31dfb0ea'}, {'device': 'watchdog', 'specParams': {'action': 'poweroff', 'model': 'ib700'}, 'type': 'watchdog', 'deviceId': '00000000-0000-0000-0000-000000000000'}], 'smp': '2', 'vmType': 'kvm', 'memSize': 8182, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', 'smpCoresPerSocket': '1', 'vmName': 'vm', 'display': 'qxl', 'nice': '0'}}
Thread-964::INFO::2013-08-15 14:57:26,105::libvirtvm::1453::vm.Vm::(_run) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::VM wrapper has started

Thread-964::WARNING::2013-08-15 14:57:26,106::vm::457::vm.Vm::(getConfDevices) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::Unknown type found, device: '{'device': 'watchdog', 'specParams': {'action': 'poweroff', 'model': 'ib700'}, 'type': 'watchdog', 'deviceId': '00000000-0000-0000-0000-000000000000'}' found


...
		<watchdog type="watchdog"/>
	</devices>
...

Thread-964::ERROR::2013-08-15 14:57:26,172::vm::704::vm.Vm::(_startUnderlyingVm) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 664, in _startUnderlyingVm
  File "/usr/share/vdsm/libvirtvm.py", line 1535, in _run
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 104, in wrapper
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2645, in createXML
libvirtError: internal error watchdog must contain model name
Thread-964::DEBUG::2013-08-15 14:57:26,175::vm::1092::vm.Vm::(setDownStatus) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::Changed state to Down: internal error watchdog must contain model name
Comment 19 Michal Skrivanek 2013-09-05 08:50:58 EDT
Martin, pls check also to regard with your latest patch in this area
Comment 20 Michal Skrivanek 2013-09-11 07:15:44 EDT
- specify versions
- do not reopen bugs on different issue. It only causes chaos and delays

thanks
Comment 21 Lukas Svaty 2013-10-02 05:14:03 EDT
tried reproducing this in version is17, could not reproduce adding watchdog without model or/and action via admin portal/rest api

libvirt-0.10.2-27.el6.x86_64 (host)

I suggest setting it to closed-current release
Comment 22 Michal Skrivanek 2013-10-02 05:45:00 EDT
thanks for checking it!

Note You need to log in before you can comment on or make changes to this bug.