Created attachment 950343 [details] log_collector Description of problem: cannot put link down on vNIC via GUI Following error appears: Error while executing action Edit VM Interface properties: Unexpected exception Version-Release number of selected component (if applicable): Red Hat Enterprise Virtualization Manager Version: 3.5.0-0.17.beta.el6ev vdsm-4.16.7.1-1.el7.x86_64 vdsm-jsonrpc-4.16.7.1-1.el7.noarch How reproducible: 100% Steps to Reproduce: RHEL7 host over JSON RPC 1. Create new VM and run it 2. Virtual Machines -> your VM -> Network Interfaces -> nic1 -> edit 3. change link state to down and click OK Actual results: link down fails Expected results: link down succeeds Additional info: engine.log 2014-10-24 13:30:18,467 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.UpdateVmInterfaceVDSCommand] (ajp-/127.0.0.1:8702-3) [1a58b1d4] Unexpected return value: StatusForXmlRpc [mCode=-32603, mMessage='VM' object has no attribute 'updateDevice'] vdsm.log Thread-895::ERROR::2014-10-24 13:30:17,587::__init__::491::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 486, in _serveRequest res = method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 264, in _dynamicMethod fn = getattr(api, methodName) AttributeError: 'VM' object has no attribute 'updateDevice'
Created attachment 950344 [details] screenshot1
Failed to put link down on vNIC. I'm not sure this is the same bug, i will attach relevant logs. I will put this BZ back to assign, if it's other bug, please let me know and i will open new BZ for that. vdsm.log Thread-22::ERROR::2014-12-16 09:31:35,333::sampling::475::vm.Vm::(collect) vmId=`5d033bad-23a9-45e6-b7cd-c517483c7f48`::Stats function failed: <AdvancedStatsFunction _sampleCpuTune at 0x2c0b5e0> Traceback (most recent call last): File "/usr/share/vdsm/virt/sampling.py", line 471, in collect statsFunction() File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__ retValue = self._function(*args, **kwargs) File "/usr/share/vdsm/virt/vm.py", line 349, in _sampleCpuTune infos = self._vm._dom.schedulerParameters() File "/usr/share/vdsm/virt/vm.py", line 689, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2134, in schedulerParameters if ret is None: raise libvirtError ('virDomainGetSchedulerParameters() failed', dom=self) libvirtError: Unable to read from '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dMic2.scope/cpu.shares': No such file or directory engine.log 2014-12-16 09:14:07,182 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.UpdateVmInterfaceVDSCommand] (ajp-/127.0.0.1:8702-7) [2f2db86c] Failed in UpdateVmInterfaceVDS method 2014-12-16 09:14:07,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.UpdateVmInterfaceVDSCommand] (ajp-/127.0.0.1:8702-7) [2f2db86c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.UpdateVmInterfaceVDSComman d return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=56, mMessage=Device instance for device identified by alias not found]] 2014-12-16 09:14:07,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.UpdateVmInterfaceVDSCommand] (ajp-/127.0.0.1:8702-7) [2f2db86c] HostName = pink-vds3.qa.lab.tlv.redhat.com 2014-12-16 09:14:07,183 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.UpdateVmInterfaceVDSCommand] (ajp-/127.0.0.1:8702-7) [2f2db86c] Command UpdateVmInterfaceVDSCommand(HostName = pink-vds3.qa.lab.tlv.redhat.c om, HostId = 9bf7ba28-cf75-47f6-95ca-3ea3084194a4, vm.vm_name=Mic2, nic=nic1 {id=3727c49e-56d5-4d50-981d-df65e3f53efe, vnicProfileId=813a636b-f52f-4803-8ae6-b5f1573f3a67, speed=1000, type=3, macAddress=00:1a:4a:16 :88:5f, linked=false, vmId=5d033bad-23a9-45e6-b7cd-c517483c7f48, vmTemplateId=null}, vmDevice=VmDevice {vmId=5d033bad-23a9-45e6-b7cd-c517483c7f48, deviceId=3727c49e-56d5-4d50-981d-df65e3f53efe, device=bridge, type =INTERFACE, bootOrder=0, specParams={outbound={peak=1280, burst=204800, average=1280}, inbound={peak=640, burst=204800, average=640}}, address=, managed=true, plugged=true, readOnly=false, deviceAlias=, customProp erties={}, snapshotId=null, logicalName=null}) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to UpdateVmInterfaceVDS, error = Device instance for device identified by alias not found, code = 56 2014-12-16 09:14:07,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.UpdateVmInterfaceVDSCommand] (ajp-/127.0.0.1:8702-7) [2f2db86c] FINISH, UpdateVmInterfaceVDSCommand, log id: 5a3be9e6 2014-12-16 09:14:07,183 ERROR [org.ovirt.engine.core.bll.network.vm.UpdateVmInterfaceCommand] (ajp-/127.0.0.1:8702-7) [2f2db86c] Command org.ovirt.engine.core.bll.network.vm.UpdateVmInterfaceCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to UpdateVmInterfaceVDS, error = Device instance for devic e identified by alias not found, code = 56 (Failed with error UpdateDevice and code 56) 2014-12-16 09:14:07,186 INFO [org.ovirt.engine.core.bll.network.vm.UpdateVmInterfaceCommand] (ajp-/127.0.0.1:8702-7) [2f2db86c] Command [id=f4d8c77a-a2e7-4464-9c48-401d98c92ae4]: Compensating DELETED_OR_UPDATED_E NTITY of org.ovirt.engine.core.common.businessentities.network.VmNic; snapshot: id=3727c49e-56d5-4d50-981d-df65e3f53efe. 2014-12-16 09:14:07,253 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [2f2db86c] Correlation ID: 2f2db86c, Call Stack: null, Custom Event ID: -1, Message: Fai led to update Interface nic1 (VirtIO) for VM Mic2. (User: admin) 2014-12-16 09:18:44,848 INFO [org.ovirt.engine.core.bll.network.vm.UpdateVmInterfaceCommand] (ajp-/127.0.0.1:8702-11) [7ef44ad7] Running command: UpdateVmInterfaceCommand internal: false. Entities affected : ID: 038dd653-dc16-48df-a06b-40338a7c98f3 Type: VMAction group CONFIGURE_VM_NETWORK with role type USER Tested on and with: vdsm-jsonrpc-java-1.0.12-1.el6ev.noarch 3.5.0-0.25.el6ev vdsm-4.16.8.1-3.el7ev.x86_64 vdsm-jsonrpc-4.16.8.1-3.el7ev.noarch RHEL7 host
Created attachment 969437 [details] fail to put link down Vnic-fail QA
For more details please contact me, i will leave my setup the way it's now.
It is different issue and by seeing it we know that the fix worked. Original issue was that the api failed due to naming mismatch. It would be good idea to open new BZ for it.
on vdsm log I see the following: Traceback (most recent call last): File "/usr/share/vdsm/virt/sampling.py", line 471, in collect statsFunction() File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__ retValue = self._function(*args, **kwargs) File "/usr/share/vdsm/virt/vm.py", line 349, in _sampleCpuTune infos = self._vm._dom.schedulerParameters() File "/usr/share/vdsm/virt/vm.py", line 689, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2134, in schedulerParameters if ret is None: raise libvirtError ('virDomainGetSchedulerParameters() failed', dom=self) libvirtError: Unable to read from '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dMic2.scope/cpu.shares': No such file or directory This seems like a a virt issue
CPU tuning -> SLA
Based on comment 6, the original issue is no longer relevant and this should be closed. We will open a different BZ for libvirt to focus on the relevant cgroup issue. For now you can disable CPU QoS and close this issue.
Doron, I have no CPU Qos enabled. Only network Qos on this Vnic. So you still can't put link down to Vnic that has Qos. All relevant logs attached, not only comment 6.
The failing call is called as part of our statistics collector. Can't be disabled. I am going to clone this bug so we can track the original network issue and the new libvirt issue separately.
Adding some more info; The root cause is documented as Bug 1139223 in RHEL 7. Indeed it has nothing to do with QoS other than the fact that we use a related API. The same goes for networking which is a completely different API and unrelated to the current issue. So we're now blocked on 1139223 and this should be verified once there's a new libvirt version with a relevant fix.
Moving back to Piotr and to ON_QA so this bug is used to track only the networking issue. That issue seems to be resolved according to the comment here: https://bugzilla.redhat.com/show_bug.cgi?id=1175234#c6
still fail with error: vdsm.log Thread-20::ERROR::2014-12-24 15:21:27,712::sampling::475::vm.Vm::(collect) vmId=`5d033bad-23a9-45e6-b7cd-c517483c7f48`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x20f9750> Traceback (most recent call last): File "/usr/share/vdsm/virt/sampling.py", line 471, in collect statsFunction() File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__ retValue = self._function(*args, **kwargs) File "/usr/share/vdsm/virt/vm.py", line 303, in _sampleCpu cpuStats = self._vm._dom.getCPUStats(True, 0) File "/usr/share/vdsm/virt/vm.py", line 689, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2016, in getCPUStats if ret is None: raise libvirtError ('virDomainGetCPUStats() failed', dom=self) libvirtError: unable to get cpu account: Operation not permitted Thread-20::DEBUG::2014-12-24 15:21:27,760::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 38 edom: 54 level: 2 message: Unable to read from '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dMic2.scope/cpu.shares': No such file or directory Thread-20::ERROR::2014-12-24 15:21:27,761::sampling::475::vm.Vm::(collect) vmId=`5d033bad-23a9-45e6-b7cd-c517483c7f48`::Stats function failed: <AdvancedStatsFunction _sampleCpuTune at 0x20f45a8> Traceback (most recent call last): File "/usr/share/vdsm/virt/sampling.py", line 471, in collect statsFunction() File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__ retValue = self._function(*args, **kwargs) File "/usr/share/vdsm/virt/vm.py", line 349, in _sampleCpuTune infos = self._vm._dom.schedulerParameters() File "/usr/share/vdsm/virt/vm.py", line 689, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2134, in schedulerParameters if ret is None: raise libvirtError ('virDomainGetSchedulerParameters() failed', dom=self) libvirtError: Unable to read from '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dMic2.scope/cpu.shares': No such file or directory libvirt.log 2014-12-24 13:22:43.057+0000: 1970: error : virFileReadAll:1208 : Failed to open file '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dMic2.scope/cpu.shares': No such file or directory 2014-12-24 13:22:43.057+0000: 1970: error : virCgroupGetValueStr:697 : Unable to read from '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2dMic2.scope/cpu.shares': No such file or directory tested on - 3.5.0-0.27.el6ev systemd-208-11.el7_0.5.x86_64
From BZ 1139223 it seems that the fix is made in version- systemd-208-20.el7
well, that's a RHEL 7.1 package, not even released yet. Hence the bug shouldn't be ON_QA
I don't understand your question, Michal - QA's job is to test not-yet-released packages. It seems that we urgently need a 7.0.z clone of systemd bug 1139223.
(In reply to Dan Kenigsberg from comment #16) > I don't understand your question, Michal - QA's job is to test > not-yet-released packages. > > It seems that we urgently need a 7.0.z clone of systemd bug 1139223. well, exactly. But we also need to be careful as if it works with systemd-208-11.el7_0.5.x86_64 it doesn't mean the bug is fixed for RHEV 3.5 as we need 7.0 package and it needs to be released in a z-stream (or we need to add Requires:). So ON_QA is a bit misleading...
Verified on - 3.5.0-0.31.el6ev with systemd-208-11.el7_0.6.x86_64