Bug 1267565 – smart state analysis for vmware vm fails with "Unable to mount filesystem. Reason:[FFI::VixDiskLib::ApiWrapper#open (errcode=15 - VIX_E_FILE_ALREADY_LOCKED): The file is already in use "

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1267565 - smart state analysis for vmware vm fails with "Unable to mount filesystem. Reason:[FFI::VixDiskLib::ApiWrapper#open (errcode=15 - VIX_E_FILE_ALREADY_LOCKED): The file is already in use "

Summary:	smart state analysis for vmware vm fails with "Unable to mount filesystem. R...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat CloudForms Management Engine
Classification:	Red Hat
Component:	SmartState Analysis (Show other bugs)
Sub Component:
Version:	5.5.0
Hardware:	Unspecified Unspecified

Priority	high Severity high
Target Milestone:	GA
Target Release:	5.5.0
Assigned To:	Rich Oliveri
QA Contact:	Aziza Karol
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2015-09-30 08:15 EDT by Aziza Karol
Modified:	2015-12-08 08:33 EST (History)
CC List:	7 users (show)

See Also:	1274075
Fixed In Version:	5.5.0.12
Doc Type:	Bug Fix
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2015-12-08 08:33:41 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
screenshot (307.30 KB, image/png) 2015-11-04 22:57 EST, Dave Johnson	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2015:2551	normal	SHIPPED_LIVE	Moderate: CFME 5.5.0 bug fixes and enhancement update	2015-12-08 12:58:09 EST

Groups: None (edit)

Comment 4 Aziza Karol 2015-10-12 08:45:00 EDT

credentials provided.

Comment 5 Dave Johnson 2015-10-16 11:24:06 EDT

Hmmmm, seeing this one now with 5.5.0.5.  This appears to show that its using the datastore value when trying to connect to the host.  That isn't going to work.  I need to check 5.5.0.6 though, stay tuned.

[----] E, [2015-10-16T11:16:17.330760 #2891:35f994] ERROR -- : Q-task_id([c7753e48-7418-11e5-8ea5-005056964f8b]) MIQ(VmScan#process_abort) job aborting, Unable to mount filesystem.  Reason:[FFI::VixDiskLib::ApiWrapper#open (errcode=18000 - VIX_E_CANNOT_CONNECT_TO_HOST): Cannot connect to the host - [cfme-esx-glob-na01a-s]

Comment 7 Dave Johnson 2015-11-04 22:57 EST

Created attachment 1089941 [details]
screenshot

Still seeing this on some VMs with 5.5.0.9, 
Unable to mount filesystem. Reason:[FFI::VixDiskLib::ApiWrapper#open (errcode=15 - VIX_E_FILE_ALREADY_LOCKED): The file is already in use - [cfme-esx-55-02-local] nachandr-f22-2/nachandr-f22-2.vmdk]

seeing some other errors as well, 
Unable to mount filesystem. Reason:[FFI::VixDiskLib::ApiWrapper#open (errcode=1 - VIX_E_FAIL): Thin/TBZ/Sparse disks cannot be opened in multiwriter mode. - [cfme-esx-55-na01b] admin-templater/admin-templater-000001.vmdk]

Comment 11 Jerry Keselman 2015-11-20 10:18:49 EST

Reassigning to Rich as he has a possible fix.

Comment 12 CFME Bot 2015-11-20 16:25:40 EST

https://github.com/ManageIQ/manageiq/pull/5549

Comment 13 CFME Bot 2015-11-23 11:20:24 EST

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/2d29b12061459b550028c2885aa9414ae71ac39e

commit 2d29b12061459b550028c2885aa9414ae71ac39e
Author:     Richard Oliveri <roliveri@redhat.com>
AuthorDate: Fri Nov 20 16:13:02 2015 -0500
Commit:     Richard Oliveri <roliveri@redhat.com>
CommitDate: Fri Nov 20 16:13:02 2015 -0500

    Fix scan snapshot creation for VMware.
    
    Issues in the previously refactored code prevent scan snapshots from
    being created for VMware VMs. This, addresses those issues.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1267565

 .../vm_or_template_shared/scanning.rb              |  1 +
 .../VolumeManager/VolMgrPlatformSupportLinux.rb    | 79 ++++------------------
 2 files changed, 16 insertions(+), 64 deletions(-)

Comment 14 CFME Bot 2015-11-23 16:12:06 EST

New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=07eb79cfa343fd5e6f7670b3f9adb272f6e3395d

commit 07eb79cfa343fd5e6f7670b3f9adb272f6e3395d
Author:     Richard Oliveri <roliveri@redhat.com>
AuthorDate: Fri Nov 20 16:13:02 2015 -0500
Commit:     Richard Oliveri <roliveri@redhat.com>
CommitDate: Mon Nov 23 11:46:37 2015 -0500

    Fix scan snapshot creation for VMware.
    
    Issues in the previously refactored code prevent scan snapshots from
    being created for VMware VMs. This, addresses those issues.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1267565

 .../vm_or_template_shared/scanning.rb              |  1 +
 .../VolumeManager/VolMgrPlatformSupportLinux.rb    | 79 ++++------------------
 2 files changed, 16 insertions(+), 64 deletions(-)

Comment 15 CFME Bot 2015-11-23 16:12:25 EST

New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=e3a6ea0d12388c7d02e832409a318d13279d823b

commit e3a6ea0d12388c7d02e832409a318d13279d823b
Merge: fa2fd37 07eb79c
Author:     Oleg Barenboim <obarenbo@redhat.com>
AuthorDate: Mon Nov 23 15:07:02 2015 -0500
Commit:     Oleg Barenboim <obarenbo@redhat.com>
CommitDate: Mon Nov 23 15:07:02 2015 -0500

    Merge branch 'vmware_snapshot_fix_55z' into '5.5.z'
    
    Fix scan snapshot creation for VMware.
    
    Issues in the previously refactored code prevent scan snapshots from
    being created for VMware VMs. This, addresses those issues.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1267565
    
    Upstream PR: https://github.com/ManageIQ/manageiq/pull/5549
    
    Clean cherry-pick.
    
    
    See merge request !520

 .../vm_or_template_shared/scanning.rb              |  1 +
 .../VolumeManager/VolMgrPlatformSupportLinux.rb    | 79 ++++------------------
 2 files changed, 16 insertions(+), 64 deletions(-)

Comment 16 Aziza Karol 2015-11-25 08:30:54 EST

Smart state analysis fails on 5.5.0.12 with the following error :

"Unable to mount filesystem. Reason:[FFI::VixDiskLib::ApiWrapper#open (errcode=18000 - VIX_E_CANNOT_CONNECT_TO_HOST): Cannot connect to the host - [cfme-netapp2554-vs1] aziza-cfme-55010-dnd/aziza-cfme-55010-dnd.vmdk FFI::VixDiskLib::ApiWrapper#open (errcode=18000 - VIX_E_CANNOT_CONNECT_TO_HOST): Cannot connect to the host - [cfme-netapp2554-vs1] aziza-cfme-55010-dnd/aziza-cfme-55010-dnd_1.vmdk]"


vim.log:
[----] W, [2015-11-25T08:22:51.802200 #22566:1d33b2c]  WARN -- : VMware(VixDiskLib): VixDiskLib: Invalid configuration file parameter.  Failed to read configuration file.
[----] I, [2015-11-25T08:22:51.814061 #22566:1d33b2c]  INFO -- : VMware(VixDiskLib): OBJLIB-LIB: Objlib initialized.
[----] I, [2015-11-25T08:22:51.814284 #22566:1d33b2c]  INFO -- : VMware(VixDiskLib): VixDiskLib: Advanced transport module not loaded.
[----] I, [2015-11-25T08:22:51.814413 #22566:1d33b2c]  INFO -- : VMware(VixDiskLib): VixDiskLib: Using transport modes from DiskLib: file:nbdssl:nbd.
[----] I, [2015-11-25T08:22:51.814531 #22566:1d33b2c]  INFO -- : VMware(VixDiskLib): VMware VixDiskLib (5.5) Release build-2962804
[----] I, [2015-11-25T08:22:51.826660 #22566:1d33b2c]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_Init: Initialization is completed.
[----] I, [2015-11-25T08:22:51.827486 #22566:1d33b2c]  INFO -- : VdlWrapper.connect: 
[----] I, [2015-11-25T08:22:51.827589 #22566:1d33b2c]  INFO -- : VdlConnection.initialize: 
[----] I, [2015-11-25T08:22:51.827821 #22566:1d33b2c]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_Connect: Establish connection.
[----] I, [2015-11-25T08:22:51.853591 #22566:1d2fcc0]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_OpenEx: Open a disk.
[----] I, [2015-11-25T08:22:51.853776 #22566:1d2fcc0]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [cfme-netapp2554-vs1] aziza-cfme-55010-dnd/aziza-cfme-55010-dnd.vmdk.
[----] I, [2015-11-25T08:22:51.853956 #22566:1d2fcc0]  INFO -- : VMware(VixDiskLib): VThreadBase detected multiple threads.
[----] I, [2015-11-25T08:22:51.918284 #22566:1d2fcc0]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: Error 18000 (listener error GVmomiFaultInvalidResponse).
[----] W, [2015-11-25T08:22:51.918475 #22566:1d2fcc0]  WARN -- : VMware(VixDiskLib): VixDiskLibVim: Login failure. Callback error 18000 at 2599.
[----] I, [2015-11-25T08:22:51.918533 #22566:1d2fcc0]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: Failed to load license manager.
[----] I, [2015-11-25T08:22:51.918836 #22566:1d2fcc0]  INFO -- : VMware(VixDiskLib): VixDiskLib: Error occurred when obtaining NFC ticket for: [cfme-netapp2554-vs1] aziza-cfme-55010-dnd/aziza-cfme-55010-dnd.vmdk. Error 18000 at 1456.
#<VixDiskLibError: FFI::VixDiskLib::ApiWrapper#open (errcode=18000 - VIX_E_CANNOT_CONNECT_TO_HOST): Cannot connect to the host>
/opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api_wrapper.rb:446:in `check_error'
/opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api_wrapper.rb:270:in `open'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:189:in `initialize'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:136:in `new'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:136:in `block in getDisk'
/opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:233:in `block in sync_synchronize'
/opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:230:in `handle_interrupt'
/opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:230:in `sync_synchronize'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:133:in `getDisk'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1624:in `perform_without_block'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1584:in `perform'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1657:in `block (2 levels) in main_loop'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1653:in `loop'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1653:in `block in main_loop'
[----] I, [2015-11-25T08:22:51.922718 #22566:1d2e4ec]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_OpenEx: Open a disk.
[----] I, [2015-11-25T08:22:51.922839 #22566:1d2e4ec]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: VixDiskLibVim_GetNfcTicket: Get NFC ticket for [cfme-netapp2554-vs1] aziza-cfme-55010-dnd/aziza-cfme-55010-dnd_1.vmdk.
[----] I, [2015-11-25T08:22:51.972796 #22566:1d2e4ec]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: Error 18000 (listener error GVmomiFaultInvalidResponse).
[----] W, [2015-11-25T08:22:51.973086 #22566:1d2e4ec]  WARN -- : VMware(VixDiskLib): VixDiskLibVim: Login failure. Callback error 18000 at 2599.
[----] I, [2015-11-25T08:22:51.973195 #22566:1d2e4ec]  INFO -- : VMware(VixDiskLib): VixDiskLibVim: Failed to load license manager.
[----] I, [2015-11-25T08:22:51.973403 #22566:1d2e4ec]  INFO -- : VMware(VixDiskLib): VixDiskLib: Error occurred when obtaining NFC ticket for: [cfme-netapp2554-vs1] aziza-cfme-55010-dnd/aziza-cfme-55010-dnd_1.vmdk. Error 18000 at 1456.
#<VixDiskLibError: FFI::VixDiskLib::ApiWrapper#open (errcode=18000 - VIX_E_CANNOT_CONNECT_TO_HOST): Cannot connect to the host>
/opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api_wrapper.rb:446:in `check_error'
/opt/rh/cfme-gemset/gems/ffi-vix_disk_lib-1.0.2/lib/ffi-vix_disk_lib/api_wrapper.rb:270:in `open'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:189:in `initialize'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:136:in `new'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:136:in `block in getDisk'
/opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:233:in `block in sync_synchronize'
/opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:230:in `handle_interrupt'
/opt/rh/rh-ruby22/root/usr/share/ruby/sync.rb:230:in `sync_synchronize'
/var/www/miq/vmdb/gems/pending/VixDiskLib/vdl_wrapper.rb:133:in `getDisk'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1624:in `perform_without_block'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1584:in `perform'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1657:in `block (2 levels) in main_loop'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1653:in `loop'
/opt/rh/rh-ruby22/root/usr/share/ruby/drb/drb.rb:1653:in `block in main_loop'
[----] I, [2015-11-25T08:22:52.173939 #22566:1d2af04]  INFO -- : VdlConnection.disconnect: 
[----] I, [2015-11-25T08:22:52.174099 #22566:1d2af04]  INFO -- : VdlWrapper.__disconnect__: 
[----] I, [2015-11-25T08:22:52.174258 #22566:1d2af04]  INFO -- : VMware(VixDiskLib): VixDiskLib: VixDiskLib_Disconnect: Disconnect.
[----] I, [2015-11-25T08:22:52.808434 #22566:111d98c]  INFO -- : startup has happened, shutdown flag is true
[----] I, [2015-11-25T08:22:52.808561 #22566:111d98c]  INFO -- : Shutting Down VixDiskLibServer
[----] I, [2015-11-25T08:22:52.808662 #22566:111d98c]  INFO -- : VixDiskLib has exited cleanly
[----] I, [2015-11-25T08:22:52.808703 #22566:111d98c]  INFO -- : VdlWrapper.__exit__ finished
[----] I, [2015-11-25T08:22:52.808999 #22566:111d98c]  INFO -- : Finished shutting down DRb
[----] I, [2015-11-25T08:22:52.809080 #22566:111d98c]  INFO -- : Service has stopped

Comment 17 Dave Johnson 2015-12-01 08:50:49 EST

Moving to verified, I believe this is related to vddk v5.5.5 issues and will open a separate BZ.  Using vddk v5.5.4 and below seems to be fine.

Comment 18 Dave Johnson 2015-12-01 08:58:47 EST

bug 1287098 opened to cover comment 17

Comment 20 errata-xmlrpc 2015-12-08 08:33:41 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2551

Note You need to log in before you can comment on or make changes to this bug.