Bug 1432679

Summary:	Smartstate Analysis Error Unable to mount filesystem Unable to determine port used by VixDiskLib VMware
Product:	Red Hat CloudForms Management Engine	Reporter:	myoder
Component:	SmartState Analysis	Assignee:	Jerry Keselman <jkeselma>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Satyajit Bulage <sbulage>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.7.0	CC:	cpelland, gblomqui, jhardy, jkeselma, obarenbo, roliveri, sbulage, simaishi, vestival
Target Milestone:	GA	Keywords:	TestOnly
Target Release:	5.9.0
Hardware:	All
OS:	All
Whiteboard:	smartstate:vmware:vm
Fixed In Version:	5.9.0.1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1441726 1441727 (view as bug list)		Environment:
Last Closed:	2018-03-06 14:50:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	Bug
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	VMware	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1441726, 1441727

Description myoder 2017-03-15 23:34:35 UTC

Description of problem:

This is after a recent update to vcenter 6.5 and updated vddk on all CF systems (VMware-vix-disklib-6.5.0-4604867.x86_64.tar.gz).  Requested to run Smart State Analysis on a single guest and recieved this error.

Unable to mount filesystem. Reason:[ERROR: VixDiskLib.connect() Unable to determine port used by VixDiskLib Server.]
smart state and smart proxy are enabled.

Stacktrace:

[----] I, [2017-03-15T14:42:19.781410 #15808:c6d130]  INFO -- : Loading disk files for VM [[sea01_dsc_pool02_31] ajen01sea01us/ajen01sea01us.vmx]
[----] I, [2017-03-15T14:42:19.817223 #15808:c6d130]  INFO -- : Snapshot create pre-check skipped for Datastore <sea01_dsc_pool02_31> due to Percentage:<>.  Space Free:<566625304576>  Disk size:<48318382080>
[----] I, [2017-03-15T14:42:20.850946 #15808:c6d130]  INFO -- : VolMgrPlatformSupportLinux.preMount: VM snapshot created in [1.058998264] seconds
[----] I, [2017-03-15T14:42:20.851291 #15808:c6d130]  INFO -- : ThumbPrint.initialize(aex03-sea01.prod.auction.local)
[----] I, [2017-03-15T14:42:20.851612 #15808:c6d130]  INFO -- : MiqVimVdlVcConnectionMod.vdlVcConnection: server - aex03-sea01.prod.auction.local
[----] I, [2017-03-15T14:42:20.893514 #15808:c6d130]  INFO -- : ThumbPrint.to_sha1 for host aex03-sea01.prod.auction.local
[----] I, [2017-03-15T14:42:20.893863 #15808:c6d130]  INFO -- : ThumbPrint.to_sha1 for host aex03-sea01.prod.auction.local is B1:25:3C:B5:83:9F:A1:9B:CD:96:B1:DC:E5:6B:EC:BE:F7:F3:91:9C
[----] E, [2017-03-15T14:42:21.081777 #15808:c6d130] ERROR -- : Unable to mount filesystem.  Reason:[ERROR: VixDiskLib.connect() Unable to determine port used by VixDiskLib Server.] for VM [[sea01_dsc_pool02_31]
 ajen01sea01us/ajen01sea01us.vmx]
[----] E, [2017-03-15T14:42:21.082031 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:141:in `get_uri'
[----] E, [2017-03-15T14:42:21.082081 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:125:in `start_service'
[----] E, [2017-03-15T14:42:21.082190 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:42:in `block in connect'
[----] E, [2017-03-15T14:42:21.082225 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/sync.rb:234:in `block in sync_synchronize'
[----] E, [2017-03-15T14:42:21.082256 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/sync.rb:231:in `handle_interrupt'
[----] E, [2017-03-15T14:42:21.082286 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/sync.rb:231:in `sync_synchronize'
[----] E, [2017-03-15T14:42:21.082315 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:39:in `connect'
[----] E, [2017-03-15T14:42:21.082344 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VMwareWebService/MiqVimVdlMod.rb:40:in `vdlVcConnection'
[----] E, [2017-03-15T14:42:21.082373 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:95:in `block in openDisks'
[----] E, [2017-03-15T14:42:21.082402 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:79:in `each'
[----] E, [2017-03-15T14:42:21.082431 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:79:in `openDisks'
[----] E, [2017-03-15T14:42:21.082460 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:67:in `vmDisks'
[----] E, [2017-03-15T14:42:21.082489 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:173:in `volumeManager'
[----] E, [2017-03-15T14:42:21.082518 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:167:in `rootTrees'
[----] E, [2017-03-15T14:42:21.082547 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/metadata/MIQExtract/MIQExtract.rb:58:in `initialize'
[----] E, [2017-03-15T14:42:21.082576 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/mixins/scanning_mixin.rb:236:in `new'
[----] E, [2017-03-15T14:42:21.082605 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/mixins/scanning_mixin.rb:236:in `scan_via_miq_vm'
[----] E, [2017-03-15T14:42:21.082644 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/manageiq/providers/vmware/infra_manager/vm_or_template_shared/scanning.rb:29:in `perform_metadata_scan'
[----] E, [2017-03-15T14:42:21.082681 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server/server_smart_proxy.rb:125:in `scan_metadata'
[----] E, [2017-03-15T14:42:21.082712 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver'
[----] E, [2017-03-15T14:42:21.082742 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout'
[----] E, [2017-03-15T14:42:21.082771 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `block in catch'
[----] E, [2017-03-15T14:42:21.082800 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
[----] E, [2017-03-15T14:42:21.082837 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
[----] E, [2017-03-15T14:42:21.082869 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:106:in `timeout'
[----] E, [2017-03-15T14:42:21.082898 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver'
[----] E, [2017-03-15T14:42:21.082927 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message'
[----] E, [2017-03-15T14:42:21.082956 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
[----] E, [2017-03-15T14:42:21.082992 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work'
[----] E, [2017-03-15T14:42:21.083023 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `loop'
[----] E, [2017-03-15T14:42:21.083052 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'
[----] E, [2017-03-15T14:42:21.083080 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
[----] E, [2017-03-15T14:42:21.083109 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
[----] E, [2017-03-15T14:42:21.083137 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop'
[----] E, [2017-03-15T14:42:21.083166 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
[----] E, [2017-03-15T14:42:21.083195 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start'
[----] E, [2017-03-15T14:42:21.083224 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
[----] E, [2017-03-15T14:42:21.083253 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker.rb:343:in `block in start'
[----] E, [2017-03-15T14:42:21.083281 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
[----] E, [2017-03-15T14:42:21.083315 #15808:c6d130] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
[----] E, [2017-03-15T14:42:21.083345 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker.rb:341:in `start'
[----] E, [2017-03-15T14:42:21.083374 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker.rb:270:in `start_worker'
[----] E, [2017-03-15T14:42:21.083403 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker.rb:150:in `block in sync_workers'
[----] E, [2017-03-15T14:42:21.083432 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker.rb:150:in `times'
[----] E, [2017-03-15T14:42:21.083460 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker.rb:150:in `sync_workers'
[----] E, [2017-03-15T14:42:21.083489 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:52:in `block in sync_workers'
[----] E, [2017-03-15T14:42:21.083552 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
[----] E, [2017-03-15T14:42:21.083584 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
[----] E, [2017-03-15T14:42:21.083613 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
[----] E, [2017-03-15T14:42:21.083755 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server.rb:346:in `block in monitor'
[----] E, [2017-03-15T14:42:21.083798 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
[----] E, [2017-03-15T14:42:21.083830 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
[----] E, [2017-03-15T14:42:21.083866 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server.rb:346:in `monitor'
[----] E, [2017-03-15T14:42:21.083910 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server.rb:368:in `block (2 levels) in monitor_loop'
[----] E, [2017-03-15T14:42:21.083942 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
[----] E, [2017-03-15T14:42:21.083983 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
[----] E, [2017-03-15T14:42:21.084014 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server.rb:368:in `block in monitor_loop'
[----] E, [2017-03-15T14:42:21.084044 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server.rb:367:in `loop'
[----] E, [2017-03-15T14:42:21.084073 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server.rb:367:in `monitor_loop'
[----] E, [2017-03-15T14:42:21.084102 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server.rb:250:in `start'
[----] E, [2017-03-15T14:42:21.084131 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
[----] E, [2017-03-15T14:42:21.084160 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/lib/workers/evm_server.rb:92:in `start'
[----] E, [2017-03-15T14:42:21.084189 #15808:c6d130] ERROR -- : MIQExtract.new /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'qq

Version-Release number of selected component (if applicable):
vCenter 6.5
CFME 5.7.1.3
vddk WMware-vix-disklib-6.5.0-4604867

How reproducible:
always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Jerry Keselman 2017-03-15 23:44:53 UTC

VDDK 6.5.0 has not been tested with ManageIQ / CloudForms to date.  As such the code to allow this release of the library to be loaded has not been added to the 
ffi-vix_disklib gem, causing it to fail since it is unable to load 6.5.

Comment 4 Jerry Keselman 2017-03-16 01:58:02 UTC

Michael, unless there is something specific to vCenter 6.5 that will not allow it to work with VDDK 6.0 (which I would find surprising) they should be able to use VDDK 6.0 until we can role out changes to support the newer VDDK.  I am going on assumptions here, however.  Someone would need to do the work to validate those.  The changes are relatively minor but would require the revised gems to be repackaged with CFME.

Comment 5 Jerry Keselman 2017-03-17 13:46:45 UTC

Just an update Hui Song has tested VDDK 6.0 against a vCenter 6.5 and was able to successfully run an SSA.  Until we are able to deploy the updated code with support for the VDDK 6.5 I would suggest this as a proven workaround.

Comment 6 Jerry Keselman 2017-03-23 15:05:29 UTC

*** Bug 1429864 has been marked as a duplicate of this bug. ***

Comment 7 Jerry Keselman 2017-04-06 21:00:37 UTC

https://github.com/ManageIQ/ffi-vix_disk_lib/pull/9 and
https://github.com/ManageIQ/manageiq-gems-pending/pull/113
fix this problem.

Comment 10 Jerry Keselman 2017-04-07 16:28:07 UTC

Rich, No reason.  Just have to be sure to include both the ffi-vix_disk_lib gem and manageiq-gems-pending repo changes.  This can go against as many downstream releases as we like.

Comment 13 Satyajit Bulage 2017-12-22 17:23:53 UTC

Able to scan VM on vSphere6.5 using VDDK 6.5 without any error.

Verified Version: 5.9.0.12.20171205180333_3e32b3d