2014-12-03 15:24:05,585 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [39f72a40] Importing VM bugzilla as external-bugzilla, as it is running on the on Host, but does not exist in the engine.
2014-12-03 15:24:05,590 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [39f72a40] VDS::UpdateVmRunTimeInfo Error: found VM on a VDS that is not in the database!
2014-12-03 15:24:05,754 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-2) [6554d0eb] Lock Acquired to object EngineLock [exclusiveLocks= key: external-bugzilla value: VM_NAME
, sharedLocks= ]
2014-12-03 15:24:05,865 WARN  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-2) [6554d0eb] CanDoAction of action AddVmFromScratch failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__VM,BALLOON_REQUESTED_ON_NOT_SUPPORTED_ARCH,$clusterArch ppc64
2014-12-03 15:24:05,865 INFO  [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (DefaultQuartzScheduler_Worker-2) [6554d0eb] Lock freed to object EngineLock [exclusiveLocks= key: external-bugzilla value: VM_NAME
, sharedLocks= ]

it's not that big deal since the only reasonable usecase for external vm is hosted engine which is not supported on ppc, but still the flood log is not good

3.5.1 is already full with bugs (over 80), and since none of these bugs were added as urgent for 3.5.1 release in the tracker bug, moving to 3.5.2

moving to 3.5.4 due to capacity planning for 3.5.3.
if you believe this should remain in 3.5.3, please sync with pm/dev/qe and a full triple ack for it. also - ensure priority is set accordingly to the bug status.

isn't this already fixed in 3.6 by enabling the balloon again? can you confirm?

Balloon in 3.6 is enabled and works.

Fix here was enabling balloon -> log is not spammed, IIUC cluster with ppc architecture will still not show external VMs, correct or was this fixed somewhere along the line?

Regarding comment#2 I believe that after adding host to NON-selfhosted engine showing all the VMs running on it as external is a useful information for admin. (Currently supported by standard x86/64)

(In reply to Lukas Svaty from comment #8)
> Fix here was enabling balloon -> log is not spammed, IIUC cluster with ppc
> architecture will still not show external VMs, correct or was this fixed
> somewhere along the line?

why would it not show it? it should
 
> Regarding comment#2 I believe that after adding host to NON-selfhosted
> engine showing all the VMs running on it as external is a useful information
> for admin. (Currently supported by standard x86/64)

we don't have that feature yet. we only show VMs created by engine at some point.

Tested on versions:
vdsm-4.17.8-1.el7ev.noarch
rhevm 3.6.0-15

Command used to create an "external" VM on a PPC host:
# vdsClient -s 0 create /dev/null vmId=485c87ea-2875-11e5-b7ea-28d244694832 memSize=256  display=vnc kvmEnable=true  vmName=istein

485c87ea-2875-11e5-b7ea-28d244694832
        Status = WaitForLaunch
        nicModel = rtl8139,pv
        statusTime = 43555273250
        kvmEnable = true
        displayIp = 0
        pid = 0
        guestDiskMapping = {}
        vmType = kvm
        displaySecurePort = -1
        memSize = 256
        displayPort = -1
        clientIp = 
        smp = 1
        vmName = istein
        display = vnc

On web admin,
on vms tab, the external VMs are not seen at all, which is not as expected.
on hosts tab, for the host, on which the VMs are running, the number of running VMs fit the number of VMs, that were created by vdsClient command and the number of qemu processes.

Michal,
Is the external VMs visibility, described in comment 10 is as expected?

Moving back to on_qa, and qa_contact to mshira for the balloon verification.

In continue to comment 10,
to cleanup the VMs from the host,
the qemu processes were killed, but yet the number of VMs run on host, still appear on the host, as if VMs are still running, and  command "vdsClient -s 0 list table" fail:
 
# vdsClient -s 0 list table
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2997, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 400, in do_list
    response = self.s.list(True, vms)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1587, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1306, in single_request
    return self.parse_response(response)
  File "/usr/lib/python2.7/site-packages/vdsm/vdscli.py", line 43, in wrapped_parse_response
    return old_parse_response(*args, **kwargs)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1482, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 794, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.TypeError'>:cannot marshal None unless allow_none is enabled">


As a comparison, I've checked the VMs creation and kill on rhevm 3.5 host.
The number of VMs run on host had moved to 0, once the qemu processes were killed.
command "vdsClient -s 0 list table" works OK (no errors, show no VMs).
The external VMs though were still seen on UI VMs tab, and I've removed them from the UI side.

Martin,

Can you please give your thoughts on this problem?

Thanks,
Ilanit.

Created attachment 1084107 [details]
vdsm log for problem to clear the VMs from host

It seems that VDSM is fine in this regards - the VM is detected and treated as expected (even incl. balloon). I couldn't see the vdsClient reproduction (supply logs if seen again).

As seen in comment 1, the issue is in engine as it doesn't correctly assume ppc64le arch. This also causes VMs to not be destroyed in VDSM when they're stopped, hanging there until reboot.

Iv'e checked the ballooning spam issue in the engine log, and as expected it's not happening any more (see also bug 1198214 - Enable balloon device for PPC, which already verified)

tested on : 
Red Hat Enterprise Virtualization Manager Version: 3.6.0-0.18.el6

we want to address the operating system (other v other ppc) issue when adding external vm
but it's not urgent really as there is no full support for external vms...they only appear as a consequence of a setup/configuration error

As ballooning support on ppc was added, the error spam is indeed not seen in the engine log, as reported in comment 15.

But, are we sure that the problem of spamming the engine log, in case of any unsupported feature, is solved/do not exist?

Such an assurance is hard to achieve since printing those (mostly) DEBUG messages can reveal a bug or failure.
Most probably they are/can be printed within handling of discrete events fired on time-based basis or by external party.

Anyway, such a spamming is undesirable and should be investigated on per-case basis.

Verified on rhevm 3.6.1-4:

External VM added, and destroyed successfully, on a PPCle64 rhel 7.2 based host.

commands run on host:
 vdsClient -s 0 create /dev/null vmId=485c87ea-2875-11e5-b7ea-28d244694832 memSize=256  display=vnc kvmEnable=true  vmName=istein
 vdsClient -s 0 list table
 vdsClient -s 0 destroy 485c87ea-2875-11e5-b7ea-28d244694832
 vdsClient -s 0 list table