Bug 1225541

Summary:	fleecing vsphere6 vms shows unable to determine port error with vddk6
Product:	Red Hat CloudForms Management Engine	Reporter:	Dave Johnson <dajohnso>
Component:	Documentation	Assignee:	Andrew Dahms <adahms>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Red Hat CloudForms Documentation <cloudforms-docs>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.4.0	CC:	adahms, cpelland, dlackey, jhardy, jkeselma, mfeifer, mwelk, obarenbo, tcarlin
Target Milestone:	GA
Target Release:	5.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	5.5.0.6	Doc Type:	Known Issue
Doc Text:	At current, attempting to perform a Smart State analysis of a VMware vSphere 6 virtual machine with Virtual Disk Development Kit 6.0 returns an 'Unable to determine port' error, causing the analysis to fail. This occurs because this version of the vddk is not installed in the directory where CFME expects it. By installing vddk6 in the appropriate directory, this is resolved.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-08 10:37:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dave Johnson 2015-05-27 16:08:50 UTC

Description of problem:
============================
Finally set this up and fleecing a vsphere6 vm with vddk6.0.0 shows the 'unable to determine port' error.  I reached out to Jerry who took a quick peek and believes the ffi code needs to be updated to support version 6


Version-Release number of selected component (if applicable):
==============================
5.4.0.2


How reproducible:
====================
100%

Steps to Reproduce:
========================
1. deploy appliance on vsphere6
2. configure host credentials
3. install vddk 6.0.0
4. reboot
5. enable smartproxy role
6. manage vsphere6 provider
7. scan vm

Actual results:
=========================
unable to determine port


Expected results:
=========================
success, vm metadata retrieved

Comment 5 Jerry Keselman 2015-08-04 12:49:54 UTC

We are working to implement the changes necessary for vddk 6.0 in both ManageIQ and the ffi-vix_disk_lib gem, where necessary.

Comment 6 Thom Carlin 2015-10-30 18:58:49 UTC

Same error with 5.5.0.8-beta1.4.20151027164951_4ab7fea and VDDK 6.0.1

evm.log excerpt:
[----] E, [2015-10-30T14:52:27.537705 #26549:f75990] ERROR -- : Unable to mount filesystem.  Reason:[ERROR: VixDiskLib.connect() Unable to determine port used by VixDiskLib Server.] for VM [[datastore_name] vm_name/vm_name.vmx]
[----] E, [2015-10-30T14:52:27.537989 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:141:in `get_uri'
[----] E, [2015-10-30T14:52:27.538036 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:125:in `start_service'
[----] E, [2015-10-30T14:52:27.538074 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:42:in `block in connect'
[----] E, [2015-10-30T14:52:27.538151 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:233:in `block in sync_synchronize'
[----] E, [2015-10-30T14:52:27.538189 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:230:in `handle_interrupt'
[----] E, [2015-10-30T14:52:27.538228 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:230:in `sync_synchronize'
[----] E, [2015-10-30T14:52:27.538265 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLib.rb:39:in `connect'
[----] E, [2015-10-30T14:52:27.538301 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/VMwareWebService/MiqVimVdlMod.rb:40:in `vdlVcConnection'
[----] E, [2015-10-30T14:52:27.538337 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:90:in `block in openDisks'
[----] E, [2015-10-30T14:52:27.538405 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:74:in `each'
[----] E, [2015-10-30T14:52:27.538443 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:74:in `openDisks'
[----] E, [2015-10-30T14:52:27.538480 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:62:in `vmDisks'
[----] E, [2015-10-30T14:52:27.538517 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:167:in `volumeManager'
[----] E, [2015-10-30T14:52:27.538553 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/MiqVm/MiqVm.rb:161:in `rootTrees'
[----] E, [2015-10-30T14:52:27.538677 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/gems/pending/metadata/MIQExtract/MIQExtract.rb:58:in `initialize'
[----] E, [2015-10-30T14:52:27.538716 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/mixins/scanning_mixin.rb:237:in `new'
[----] E, [2015-10-30T14:52:27.538753 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/mixins/scanning_mixin.rb:237:in `scan_via_miq_vm'
[----] E, [2015-10-30T14:52:27.538789 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/manageiq/providers/vmware/infra_manager/vm_or_template_shared/scanning.rb:17:in `perform_metadata_scan'
[----] E, [2015-10-30T14:52:27.538835 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_server/server_smart_proxy.rb:123:in `scan_metadata'
[----] E, [2015-10-30T14:52:27.538874 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue.rb:345:in `block in deliver'
[----] E, [2015-10-30T14:52:27.538909 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
[----] E, [2015-10-30T14:52:27.538945 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch'
[----] E, [2015-10-30T14:52:27.539014 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
[----] E, [2015-10-30T14:52:27.539052 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
[----] E, [2015-10-30T14:52:27.539088 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout'
[----] E, [2015-10-30T14:52:27.539128 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue.rb:341:in `deliver'
[----] E, [2015-10-30T14:52:27.539164 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message'
[----] E, [2015-10-30T14:52:27.539247 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
[----] E, [2015-10-30T14:52:27.539284 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:151:in `block in do_work'
[----] E, [2015-10-30T14:52:27.539320 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `loop'
[----] E, [2015-10-30T14:52:27.539355 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `do_work'
[----] E, [2015-10-30T14:52:27.539391 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:323:in `block in do_work_loop'
[----] E, [2015-10-30T14:52:27.539427 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:320:in `loop'
[----] E, [2015-10-30T14:52:27.539462 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:320:in `do_work_loop'
[----] E, [2015-10-30T14:52:27.539498 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:142:in `run'
[----] E, [2015-10-30T14:52:27.539600 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:123:in `start'
[----] E, [2015-10-30T14:52:27.539654 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/app/models/miq_worker/runner.rb:24:in `start_worker'
[----] E, [2015-10-30T14:52:27.539692 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/lib/workers/bin/worker.rb:2:in `<top (required)>'
[----] E, [2015-10-30T14:52:27.539728 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/bundler/gems/rails-e2fcb2b4aec6/railties/lib/rails/commands/runner.rb:60:in `load'
[----] E, [2015-10-30T14:52:27.539763 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/bundler/gems/rails-e2fcb2b4aec6/railties/lib/rails/commands/runner.rb:60:in `<top (required)>'
[----] E, [2015-10-30T14:52:27.539799 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/bundler/gems/rails-e2fcb2b4aec6/railties/lib/rails/commands/commands_tasks.rb:123:in `require'
[----] E, [2015-10-30T14:52:27.539881 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/bundler/gems/rails-e2fcb2b4aec6/railties/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
[----] E, [2015-10-30T14:52:27.539919 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/bundler/gems/rails-e2fcb2b4aec6/railties/lib/rails/commands/commands_tasks.rb:90:in `runner'
[----] E, [2015-10-30T14:52:27.539955 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/bundler/gems/rails-e2fcb2b4aec6/railties/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
[----] E, [2015-10-30T14:52:27.539997 #26549:f75990] ERROR -- : MIQExtract.new /opt/rh/cfme-gemset/bundler/gems/rails-e2fcb2b4aec6/railties/lib/rails/commands.rb:17:in `<top (required)>'
[----] E, [2015-10-30T14:52:27.540035 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/bin/rails:4:in `require'
[----] E, [2015-10-30T14:52:27.540071 #26549:f75990] ERROR -- : MIQExtract.new /var/www/miq/vmdb/bin/rails:4:in `<main>'

vim.log excerpt:

ffi-vix_disk_lib: failed to load 6.0.0 version with error: Could not open library 'vixDiskLib.so.6.0.0': vixDiskLib.so.6.0.0: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.6.0.0': libvixDiskLib.so.6.0.0: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.5.4 version with error: Could not open library 'vixDiskLib.so.5.5.4': vixDiskLib.so.5.5.4: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.5.4': libvixDiskLib.so.5.5.4: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.5.2 version with error: Could not open library 'vixDiskLib.so.5.5.2': vixDiskLib.so.5.5.2: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.5.2': libvixDiskLib.so.5.5.2: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.5.1 version with error: Could not open library 'vixDiskLib.so.5.5.1': vixDiskLib.so.5.5.1: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.5.1': libvixDiskLib.so.5.5.1: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.5.0 version with error: Could not open library 'vixDiskLib.so.5.5.0': vixDiskLib.so.5.5.0: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.5.0': libvixDiskLib.so.5.5.0: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.1.3 version with error: Could not open library 'vixDiskLib.so.5.1.3': vixDiskLib.so.5.1.3: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.1.3': libvixDiskLib.so.5.1.3: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.1.2 version with error: Could not open library 'vixDiskLib.so.5.1.2': vixDiskLib.so.5.1.2: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.1.2': libvixDiskLib.so.5.1.2: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.1.1 version with error: Could not open library 'vixDiskLib.so.5.1.1': vixDiskLib.so.5.1.1: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.1.1': libvixDiskLib.so.5.1.1: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.1.0 version with error: Could not open library 'vixDiskLib.so.5.1.0': vixDiskLib.so.5.1.0: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.1.0': libvixDiskLib.so.5.1.0: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.0.4 version with error: Could not open library 'vixDiskLib.so.5.0.4': vixDiskLib.so.5.0.4: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.0.4': libvixDiskLib.so.5.0.4: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 5.0.0 version with error: Could not open library 'vixDiskLib.so.5.0.0': vixDiskLib.so.5.0.0: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.5.0.0': libvixDiskLib.so.5.0.0: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 1.2.0 version with error: Could not open library 'vixDiskLib.so.1.2.0': vixDiskLib.so.1.2.0: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.1.2.0': libvixDiskLib.so.1.2.0: cannot open shared object file: No such file or directory.
ffi-vix_disk_lib: failed to load 1.1.2 version with error: Could not open library 'vixDiskLib.so.1.1.2': vixDiskLib.so.1.1.2: cannot open shared object file: No such file or directory.
Could not open library 'libvixDiskLib.so.1.1.2': libvixDiskLib.so.1.1.2: cannot open shared object file: No such file or directory.
/opt/rh/rh-ruby22/root/usr/share/gems/gems/ffi-1.9.8/lib/ffi/library.rb:162:in `ffi_libraries': no library specified (LoadError)
	from /opt/rh/rh-ruby22/root/usr/share/gems/gems/ffi-1.9.8/lib/ffi/library.rb:240:in `attach_function'
	from /opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api.rb:98:in `<module:API>'
	from /opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api.rb:5:in `<module:VixDiskLib>'
	from /opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api.rb:4:in `<module:FFI>'
	from /opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api.rb:3:in `<top (required)>'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib.rb:7:in `<top (required)>'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api_wrapper.rb:1:in `<top (required)>'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:128:in `require'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:128:in `rescue in require'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:39:in `require'
	from /var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:3:in `<top (required)>'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /opt/rh/rh-ruby22/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /var/www/miq/vmdb/gems/pending/VixDiskLib/VixDiskLibServer.rb:11:in `<main>'

Could this be due to removal of the vddk install?

Comment 7 Thom Carlin 2015-10-30 18:59:24 UTC

Templates have the wait_for_policy error (never starts the task) which masks this error.

Comment 8 Jerry Keselman 2015-11-01 16:51:39 UTC

We did not add VDDK 6.0.1 support.  We added 6.0.0.

Comment 9 Jerry Keselman 2015-11-09 18:03:26 UTC

Thom - can you determine if this error happens with VDK 6.0.0?  We only tested against and coded for 6.0.0 at this point.  Thank you.

Comment 11 Jerry Keselman 2015-11-18 13:30:09 UTC

The problem is you have the VDDK libraries installed in the directory /usr/lib/vmware-vix-disklib-distrib/lib64 instead of /usr/lib/vmware-vix-disklib/lib64.  That is why the libraries cannot be found.  You can see on some of your other appliances, like 10.8.58.83, for example, that the library is is the correct directory and you are not getting the error of the library not being found - the VDDK is being invoked (and getting other errors that are being handled by other BZs.  I would like to close this BZ with no trouble found if you agree.

Comment 12 Jerry Keselman 2015-11-18 14:30:16 UTC

After discussion with Thom Carlin, it was determined that some end-user doc is required for this issue.  Prior to VDDK 6.0, VMware shipped an installer with the libraries that installed everything in /usr/lib/vmware-vix-disklib.  As of 6.0, the installer is no longer shipped.  We need to let the users know to install the libs in this directory or  our Smart State Analysis code for VMware will not work.

Comment 13 Marianne Feifer 2015-11-18 19:20:56 UTC

Can you tell me what steps they need to take to get the vddk installed/copied into the correct place?

Comment 14 Jerry Keselman 2015-11-18 19:59:26 UTC

A KB needs to be written describing installation of VDDK 6.0 and higher (assuming that VMware does not decide to recreate their installation script).  The info should be along the lines of:


1. Prior to release 6.0 of the VDDK VMware provided an installation script that installed the necessary files into the library directory at /usr/lib/vmware-vix-disklib.
2.  In order to install a 6.0+ release of the VDDK onto a ManageIQ appliance, do the following:
3.  Copy the downloaded gzip file to /tmp on the appliance.
4.  unzip the archive file.
5.  If /usr/lib/vmware-vix-disklib already exists, backup and remove the contents of the directory.
6.  if the directory does not exist, create it by running "mkdir /usr/lib/vmware-vix-disklib"
7.  Under the unzipped archive file in /tmp, there are several directories.  Copy the following directories and their
contents into /usr/lib/vmware-vix-disklib:
    bin64
    include
    lib64

Comment 15 Marianne Feifer 2015-11-24 13:29:29 UTC

Andrew, can you please ensure that this is in the RNs for 5.5?  This is a big issue.  I can create a KB, but would be more appropriate in a Release Note.

Comment 17 Marianne Feifer 2015-12-07 21:03:02 UTC

If this is dropped from the advisory, does that mean that the doc text doesn't show up in release notes?

Comment 18 Andrew Dahms 2015-12-08 10:37:44 UTC

This known issue has been added to the Release Notes, the doc text of which is now live on the Customer Portal.

Closing.

Comment 19 Marianne Feifer 2015-12-08 13:11:07 UTC

Thanks Andrew! ( I was bleary eyed and missed it)

Additional steps required to add symlinks:

ln -s /usr/lib/vmware-vix-disklib/lib64/libvixDiskLib.so /usr/lib/libvixDiskLib.so
ln -s /usr/lib/vmware-vix-disklib/lib64/libvixDiskLib.so.6 /usr/lib/libvixDiskLib.so.6
reboot

Comment 20 Dave Johnson 2016-01-22 22:02:14 UTC

https://access.redhat.com/articles/2078103