Bug 1384113

Summary:	provisioning two vms in one provisioning operation leads to an error during poll_clone_complete
Product:	Red Hat CloudForms Management Engine	Reporter:	Felix Dewaleyne <fdewaley>
Component:	Provisioning	Assignee:	William Fitzgerald <wfitzger>
Status:	CLOSED NOTABUG	QA Contact:	Leo Khomenko <lkhomenk>
Severity:	medium	Docs Contact:
Priority:	high
Version:	5.6.0	CC:	agrare, fdewaley, jhardy, obarenbo
Target Milestone:	GA
Target Release:	cfme-future
Hardware:	All
OS:	All
Whiteboard:	vsphere:provision
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-01-17 15:36:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	CFME Core	Target Upstream Version:
Embargoed:

Description Felix Dewaleyne 2016-10-12 15:06:40 UTC

Description of problem:
provisioning two vms in one provisioning operation leads to an error during poll_clone_complete : MIQ(ManageIQ::Providers::Vmware::InfraManager::Provision#provision_error) [[RuntimeError]: The operation is not supported on the object.] encountered during phase [poll_clone_complete]

Version-Release number of selected component (if applicable):
5.6.1.2

How reproducible:
very frequently on customer environment

Steps to Reproduce:
1. provision 2 vms from a template
2.
3.

Actual results:
the provision stops at the 2nd vm with the error 

MIQ(ManageIQ::Providers::Vmware::InfraManager::Provision#provision_error) [[RuntimeError]: The operation is not supported on the object.] encountered during phase [poll_clone_complete]

Expected results:
the provision of both vms passes.
customer also hit by memory issues but they do notseem to affect this issue directly

Additional info:
more data from the customer environment available in private notes

Comment 6 Greg McCullough 2016-10-13 11:53:50 UTC

Billy - Please work with Tina on this issue, but I think you will also want to consult with Adam as the returned error is coming from VMware.

Felix - Can you verify if this is standard provisioning or service provisioning?  Also, would like to know from the customer if they were provisioning this way in the past or did they only start provisioning multiple VMs recently?  Since the error is being raised from VMware we would want to know if any errors or recent changes within the VMware environment might be contributing to this failure.

Comment 7 William Fitzgerald 2016-10-13 19:39:41 UTC

I am seeing some broker errors in the evm.log.  

362964 [----] E, [2016-10-10T15:16:44.146271 #22526:377990] ERROR -- : MIQ(MiqFaultTolerantVim._connect) EMS: [Noida] [Broker] Unable to connect to: [innoivcs1.pbi.global.pvt] beca       use Broker is not available (connection error).
362965 [----] W, [2016-10-10T15:16:44.151091 #22526:377990]  WARN -- : MIQ(MiqVimBrokerWorker.broker_unavailable) The following error was encountered, 'Broker is not available (con       nection error).', the broker server should be restarted on the next heartbeat
362966 [----] I, [2016-10-10T15:16:44.194687 #22526:377990]  INFO -- : MIQ(MiqQueue.put) Message id: [3000173993630],  id: [], Zone: [worker noida], Role: [], Server: [b28a3e1e-110       4-11e1-89a4-005056a8007d], Ident: [miq_server], Target id: [], Instance id: [3000000000002], Task id: [], Command: [MiqServer.message_for_worker], Timeout: [600], Priority:        [100], State: [ready], Deliver On: [], Data: [], Args: [3000001045086, "broker_unavailable"]
362967 [----] W, [2016-10-10T15:16:44.195072 #22526:377990]  WARN -- : MIQ(ManageIQ::Providers::Vmware::InfraManager#with_provider_connection) Reported the broker unavailable
362968 [----] E, [2016-10-10T15:16:44.195263 #22526:377990] ERROR -- : MIQ(MiqQueue#m_callback) Message id: [3000173993615]: Broker is not available (connection error).
362969 [----] E, [2016-10-10T15:16:44.195365 #22526:377990] ERROR -- : MIQ(MiqQueue#m_callback) backtrace: /var/www/miq/vmdb/gems/pending/util/miq_fault_tolerant_vim.rb:191:in `res       cue in _connect_with_broker'

Comment 8 Adam Grare 2016-10-17 16:01:46 UTC

There seem to be two different errors here, I see "The operation is not supported on the object." on multiple appliances but the broker_unavailable error only on one.

"The operation is not supported on the object" is a general fault [0] thrown typically when:
1. Trying to run a vc-only method on a host
2. A feature is disabled due to licensing
3. An invalid object (e.g.: Folder) was selected to do an operation (e.g.: cloneVM_Task)

#3 is most likely in this case, I'll go through the clone task to see if anything stands out as invalid.

Can the customer try to clone a VM through the vCenter client with exactly the same selections and user as one of the failed provision tasks?

[0]: https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vmodl.fault.NotSupported.html

Comment 9 Felix Dewaleyne 2016-10-18 15:57:51 UTC

the broker errors are likely due to the memory issues of the environment and can be discarded. I've asked the customer, waiting for his feedback.

Comment 10 William Fitzgerald 2017-01-11 22:59:59 UTC

Felix,

Any feedback from the customer?

Comment 11 Felix Dewaleyne 2017-01-17 11:45:34 UTC

the customer's issue disappeared after application of scripts and memory changes.

Comment 12 William Fitzgerald 2017-01-17 14:10:34 UTC

Felix,

Can I close this ticket ?

Billy

Comment 13 Felix Dewaleyne 2017-01-17 15:29:31 UTC

I think yes

Comment 14 William Fitzgerald 2017-01-17 15:36:09 UTC

Thanks