Bug 725596

Summary: unmount rhevm nfs failure error; device is busy
Product: [Retired] CloudForms Cloud Engine Reporter: Dave Johnson <dajohnso>
Component: aeolus-configureAssignee: Richard Su <rwsu>
Status: CLOSED ERRATA QA Contact: Dave Johnson <dajohnso>
Severity: low Docs Contact:
Priority: unspecified    
Version: 1.0.0CC: cpelland, dajohnso, meyering, ssachdev, whayutin, zaitcev
Target Milestone: beta   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-15 20:34:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dave Johnson 2011-07-26 00:57:39 UTC
Description of problem:
I see this error sometimes, usually a re-run on cleanup fixes it.  Sometimes it is because I was trying to run cleanup while on the nfs mount.  This particular time, I ran into a issue where dc-rhev-image (push) command was hung.

If the user is running cleanup, then we probably could/should kill any push processes

Comment 1 Dave Johnson 2011-07-26 00:58:09 UTC
Forgot to post the cleanup error:

err: /Stage[main]/Aeolus::Rhevm::Disabled/Mount[/mnt/rhevm-nfs]/ensure: change from mounted to unmounted failed: Execution of '/bin/umount /mnt/rhevm-nfs' returned 16: umount.nfs: /mnt/rhevm-nfs: device is busy
umount.nfs: /mnt/rhevm-nfs: device is busy

Comment 2 wes hayutin 2011-09-28 16:40:17 UTC
making sure all the bugs are at the right version for future queries

Comment 4 Dave Johnson 2011-11-16 22:39:37 UTC
}
2011-11-16 18:13:13,432 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder pid(3777) Message: Executing external RHEV-M push command (['/usr/bin/dc-rhev-image', '/tmp/tmpQzGWlu'])
2011-11-16 18:13:13,447 DEBUG paste.httpserver.ThreadPool pid(3777) Message: Added task (0 tasks queued)

^C

[root@qeblade27 ~]# aeolus-cleanup
notice: /Stage[main]/Aeolus::Conductor::Disabled/Service[conductor-dbomatic]/ensure: ensure changed 'running' to 'stopped'
err: /Stage[main]/Aeolus::Profiles::Rhevm::Disabled/Mount[/mnt/rhevm-nfs]/ensure: change from mounted to unmounted failed: Execution of '/bin/umount /mnt/rhevm-nfs' returned 16: umount.nfs4: /mnt/rhevm-nfs: device is busy
umount.nfs4: /mnt/rhevm-nfs: device is busy


Still seeing this with yesterdays rpms

[root@qeblade27 ~]# rpm -qa | grep aeolus
aeolus-conductor-0.6.0-0.20111114172747gitde62906.fc15.noarch
rubygem-aeolus-cli-0.1.0-3.20111114114625git42b14c0.fc15.noarch
aeolus-conductor-daemons-0.6.0-0.20111114172747gitde62906.fc15.noarch
rubygem-aeolus-image-0.1.0-4.20111024205454git6b2b696.fc15.noarch
aeolus-conductor-doc-0.6.0-0.20111114172747gitde62906.fc15.noarch
aeolus-configure-2.3.0-0.20111111160331git9bde372.fc15.noarch
aeolus-all-0.6.0-0.20111114172747gitde62906.fc15.noarch

Comment 5 Richard Su 2011-12-14 01:32:31 UTC
Dave,

I'm having trouble figuring out a way to kill dc-rhev-image. A kill or kill -9 has no affect on it. 

I looked into nfs mount options and there isn't one to make nfs operations interruptible. There was intr but that has been deprecated since kernel 2.6.25.

Since the bug was opened, rhev pushes do have a specified timeout, currently at 30 mins. So that may have helped with the hung process you experienced before.

I'm going to add some iwhd folks to the cc list to see if they have any suggestions.

- Richard

Comment 6 Jim Meyering 2011-12-14 08:15:17 UTC
Hi Richard,

From what I read about the deprecation of the "intr" mount option,
we are supposed to be able to use kill -9 to interrupt a pending
NFS operation.  Since that has no effect, you may want to talk
to NFS or kernel people.

Comment 7 Richard Su 2011-12-14 20:52:44 UTC
steved said that interrupting nfs operations is not recommended because it can lead to data corruption.

I suggest we leave the it unmounted. I can add a check to the unmount operation to check if there are any dc-rhev-image processes active? If there are, don't unmount.

Dave, how does that sound to you?

Comment 8 Richard Su 2011-12-21 00:16:32 UTC
patch pushed to configure commit 83993ba03907354a552a060de3f664fd04fa1167

Comment 9 Richard Su 2011-12-23 01:25:45 UTC
Available in aeolus-configure-2.5.0-0.20111222093431gitd3ed726

Comment 11 Dave Johnson 2012-02-27 22:54:27 UTC
This one was blocked by bug 788644 but that dependency is no longer valid.  cleanup is now down using 'aeolus-cleanup -p rhevm'

good 2 go in aeolus-configure-2.5.0-15.el6.noarch

Comment 12 errata-xmlrpc 2012-05-15 20:34:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0586.html