Created attachment 777286 [details] engine.log Description of problem: Launching Vm fails. Version-Release number of selected component (if applicable): is6 How reproducible: 100% Steps to Reproduce: 1. Add Cluster with ovirt service enabled and host 2. Add new Vm 3. Launch it Actual results: Vm failed to run. Expected results: Vm should be runnable Additional info:
what VM, on what host, vdsm/libvirt logs? man, come on... anyway, error coming from libvirt - 'ebtables tool is missing', related to nwfilter -> moving to network
could not reproduce it in 3.3 compatiblity cluster today... happend again in 3.2 adding vdsm and libvirt logs host: RHEV Hypervisor - 6.4 - 20130528.0.el6_4 root@localhost : rpm -qa | grep vdsm vdsm-xmlrpc-4.10.2-22.0.el6ev.noarch vdsm-cli-4.10.2-22.0.el6ev.noarch vdsm-4.10.2-22.0.el6ev.x86_64 vdsm-hook-vhostmd-4.10.2-22.0.el6ev.noarch vdsm-reg-4.10.2-22.0.el6ev.noarch vdsm-python-4.10.2-22.0.el6ev.x86_64 root@localhost : rpm -qa | grep libvirt libvirt-lock-sanlock-0.10.2-18.el6_4.5.x86_64 libvirt-client-0.10.2-18.el6_4.5.x86_64 libvirt-python-0.10.2-18.el6_4.5.x86_64 libvirt-cim-0.6.1-4.el6.x86_64 libvirt-0.10.2-18.el6_4.5.x86_64
Given "VM vm is down. Exit message: internal error cannot create rule since ebtables tool is missing..", would you see if `ebtables` is installed on your host and executable by libvirt?
Created attachment 778786 [details] libvirtd.log adding libvirtd.log
What's your `virsh -r nwfilter-dumpxml vdsm-no-mac-spoofing`? Does it happen on any other host? The ebtables failure gives me no clue, maybe Laine can chip in: 2013-07-23 12:57:57.331+0000: 12965: debug : virCommandRunAsync:2227 : Command result 0, with PID 19313 2013-07-23 12:57:57.345+0000: 12965: debug : virCommandRun:2025 : Result exit status 255, stdout: '' stderr: 'Illegal target name 'libvirt-J-vnet0'. Illegal target name 'libvirt-P-vnet0'. Chain 'libvirt-J-vnet0' doesn't exist. Chain 'libvirt-P-vnet0' doesn't exist. Chain 'libvirt-J-vnet0' doesn't exist. Chain 'libvirt-P-vnet0' doesn't exist. Chain 'libvirt-J-vnet0' doesn't exist. Chain 'libvirt-J-vnet0' doesn't exist. Chain 'libvirt-P-vnet0' doesn't exist. Chain 'libvirt-P-vnet0' doesn't exist. Can you reproduce this with `virsh` directly with a domxml similar to <domain type="kvm"> <name>watchdog_vm0</name> <uuid>0c059292-d85d-4b83-8cb0-75fc90271ded</uuid> <memory>1048576</memory> <currentMemory>1048576</currentMemory> <vcpu>1</vcpu> <devices> <channel type="unix"> <target name="com.redhat.rhevm.vdsm" type="virtio"/> <source mode="bind" path="/var/lib/libvirt/qemu/channels/watchdog_vm0.com.redhat.rhevm.vdsm"/> </channel> <channel type="unix"> <target name="org.qemu.guest_agent.0" type="virtio"/> <source mode="bind" path="/var/lib/libvirt/qemu/channels/watchdog_vm0.org.qemu.guest_agent.0"/> </channel> <input bus="ps2" type="mouse"/> <channel type="spicevmc"> <target name="com.redhat.spice.0" type="virtio"/> </channel> <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> <channel mode="secure" name="main"/> <channel mode="secure" name="inputs"/> <channel mode="secure" name="cursor"/> <channel mode="secure" name="playback"/> <channel mode="secure" name="record"/> <channel mode="secure" name="display"/> <channel mode="secure" name="usbredir"/> <channel mode="secure" name="smartcard"/> </graphics> <sound model="ich6"/> <controller model="virtio-scsi" type="scsi"/> <video> <model heads="1" type="qxl" vram="65536"/> </video> <interface type="bridge"> <mac address="00:1a:4a:22:3f:dc"/> <model type="virtio"/> <source bridge="rhevm"/> <filterref filter="vdsm-no-mac-spoofing"/> <link state="up"/> <boot order="1"/> </interface> <memballoon model="virtio"/> <disk device="cdrom" snapshot="no" type="file"> <source file="" startupPolicy="optional"/> <target bus="ide" dev="hdc"/> <readonly/> <serial></serial> </disk> </devices> <os> <type arch="x86_64" machine="rhel6.4.0">hvm</type> <smbios mode="sysinfo"/> </os> <sysinfo type="smbios"> <system> <entry name="manufacturer">Red Hat</entry> <entry name="product">RHEV Hypervisor</entry> <entry name="version">6.4-20130528.0.el6_4</entry> <entry name="serial">4C4C4544-0034-5310-8052-B1C04F4A354A</entry> <entry name="uuid">0c059292-d85d-4b83-8cb0-75fc90271ded</entry> </system> </sysinfo> <clock adjustment="0" offset="variable"> <timer name="rtc" tickpolicy="catchup"/> </clock> <features> <acpi/> </features> <cpu match="exact"> <model>Conroe</model> <topology cores="1" sockets="1" threads="1"/> </cpu> </domain>
now running is7 at the moment couldn't reproduce this [root@localhost ~]# virsh -r nwfilter-dumpxml vdsm-no-mac-spoofing <filter name='vdsm-no-mac-spoofing' chain='root'> <uuid>2857b9e3-0f05-c413-c30d-ba143697d71d</uuid> <filterref filter='no-mac-spoofing'/> <filterref filter='no-arp-mac-spoofing'/> </filter>
can you find and attach a log-rotated libvirtd.log with the string "ebtables tool is missing"?
went through log-rotated libvirtd.log and "ebtables tool is missing" was not in found
Please reopen if it ever reproduces (in a way other than deleting /usr/sbin/ebtables from the file system).
Created attachment 786967 [details] libvirtd log of hypervisor
Created attachment 786968 [details] engine log when VM started of hypervisor
Created attachment 786969 [details] vdsm.log of hypervisor
Created attachment 786971 [details] engine log when Vm tried to start on RHEL
Created attachment 786972 [details] vdsm.log from rhel
Reproduced this bug on is10 in 3.2 cluster When set option "Start running on specific host" for VM Vm actually tried to start running on the other host Set Start Running on specific host: 10.34.62.203 - RHEL Vm Started running on RHEV-H host (10.34.62.204) and was shut down in less than 1 second logs: engine.log of this action: engine-rhevh.log vdsm.log of this action: vdsm-rhevh.log libvirtd.log of this action: libvirtd-rhevh.log When I set Start Running on specific host on 10.34.62.204 - RHEVH Vm tried runnning on 10.34.62.203 my RHEL host logs engine.log of this action: engine-rhel.log vdsm.log of this action: vdsm-rhel.log libvirtd.log of this action: nothing appeared in libvirtd.log This may be two bugs one that its starting on another host and that it is not actually starting the VM This bug is not that easy to reproduce so if somebody want to try for himself I can provide my setup while its still reproducable.
Why do you consider this reproduction as the same bug? I see a completely different error, unrelated to ebtables. Thread-963::DEBUG::2013-08-15 14:57:26,105::BindingXMLRPC::920::vds::(wrapper) return vmCreate with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.4.0', 'vmId': '3ba9eb8e-bb15-4219-800a-c80c3badb6e5', 'pid': '0', 'memGuaranteedSize': 1024, 'timeOffset': '0', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'SandyBridge', ...., 'clientIp': '', 'nicModel': 'rtl8139,pv', 'smartcardEnable': 'false', 'kvmEnable': 'true', 'transparentHugePages': 'true', 'devices': [{'device': 'qxl', 'specParams': {'vram': '65536', 'heads': '1'}, 'type': 'video', 'deviceId': '19271706-bfe1-482c-8a7d-4d6b15c9f4b3'}, {'index': '2', 'iface': 'ide', 'specParams': {'path': ''}, 'readonly': 'true', 'deviceId': 'a33c4d4e-51db-490f-a5c3-ec39214d977a', 'path': '', 'device': 'cdrom', 'shared': 'false', 'type': 'disk'}, {'index': 0, 'iface': 'virtio', 'format': 'raw', 'type': 'disk', 'volumeID': '8aa7bf39-4be2-44e5-806e-85557c575c75', 'imageID': '2cb74bce-7583-4240-8e16-e71dcfa0b4fa', 'specParams': {}, 'readonly': 'false', 'domainID': '11f24381-303a-4b86-97d9-0aa1d53e7963', 'deviceId': '2cb74bce-7583-4240-8e16-e71dcfa0b4fa', 'poolID': '18bba25b-3a1e-47d8-be04-eb54c59090b8', 'device': 'disk', 'shared': 'false', 'propagateErrors': 'off', 'optional': 'false'}, {'device': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:1a:4a:22:3f:dc', 'linkActive': 'true', 'network': 'rhevm', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': '9964bfed-f960-480f-a80f-aec13dcf26ae', 'device': 'bridge', 'type': 'interface'}, {'device': 'memballoon', 'specParams': {'model': 'virtio'}, 'type': 'balloon', 'deviceId': '53f3bb96-5fa5-405b-872b-f0cf31dfb0ea'}, {'device': 'watchdog', 'specParams': {'action': 'poweroff', 'model': 'ib700'}, 'type': 'watchdog', 'deviceId': '00000000-0000-0000-0000-000000000000'}], 'smp': '2', 'vmType': 'kvm', 'memSize': 8182, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard', 'smpCoresPerSocket': '1', 'vmName': 'vm', 'display': 'qxl', 'nice': '0'}} Thread-964::INFO::2013-08-15 14:57:26,105::libvirtvm::1453::vm.Vm::(_run) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::VM wrapper has started Thread-964::WARNING::2013-08-15 14:57:26,106::vm::457::vm.Vm::(getConfDevices) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::Unknown type found, device: '{'device': 'watchdog', 'specParams': {'action': 'poweroff', 'model': 'ib700'}, 'type': 'watchdog', 'deviceId': '00000000-0000-0000-0000-000000000000'}' found ... <watchdog type="watchdog"/> </devices> ... Thread-964::ERROR::2013-08-15 14:57:26,172::vm::704::vm.Vm::(_startUnderlyingVm) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 664, in _startUnderlyingVm File "/usr/share/vdsm/libvirtvm.py", line 1535, in _run File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 104, in wrapper File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2645, in createXML libvirtError: internal error watchdog must contain model name Thread-964::DEBUG::2013-08-15 14:57:26,175::vm::1092::vm.Vm::(setDownStatus) vmId=`3ba9eb8e-bb15-4219-800a-c80c3badb6e5`::Changed state to Down: internal error watchdog must contain model name
Martin, pls check also to regard with your latest patch in this area
- specify versions - do not reopen bugs on different issue. It only causes chaos and delays thanks
tried reproducing this in version is17, could not reproduce adding watchdog without model or/and action via admin portal/rest api libvirt-0.10.2-27.el6.x86_64 (host) I suggest setting it to closed-current release
thanks for checking it!