Bug 430606 - livecd-creator wipes out host /dev if it cannot unmount the install_root
livecd-creator wipes out host /dev if it cannot unmount the install_root
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: livecd-tools (Show other bugs)
8
All Linux
medium Severity medium
: ---
: ---
Assigned To: Jeremy Katz
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-01-28 18:44 EST by Orion Poplawski
Modified: 2008-08-01 13:20 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-04-14 21:59:46 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Orion Poplawski 2008-01-28 18:44:15 EST
Description of problem:

On my Fedora 8 system, livecd-creator is unable to unmount the install_root
properly:

 Building an initramfs at /boot/livecd-initramfs-2.6.24-2.fc9.img for kernel
2.6.24-2.fc9
Done; initramfs is 4.2M.

umount: /export/backup1/tmp/livecd-creator-wrcICh/install_root/dev: device is busy
umount: /export/backup1/tmp/livecd-creator-wrcICh/install_root/dev: device is busy
umount: /export/backup1/tmp/livecd-creator-wrcICh/install_root: device is busy
umount: /export/backup1/tmp/livecd-creator-wrcICh/install_root: device is busy
ioctl: LOOP_CLR_FD: Device or resource busy

livecd-creator then goes on to try to cleanup the directory and ends up wiping
out the hosts system's /dev/ filesystem because it is still mounted.

We need two things:

- Figure out why the filesytem is busy and cannot be unmounted.
- Abort if we can't unmount it (or at least don't try to clean up the
install_root directory)


Version-Release number of selected component (if applicable):
livecd-tools-013-1.fc8

How reproducible:
Every time
Comment 1 Orion Poplawski 2008-01-29 19:33:50 EST
Looks like this has to do with commands I'm running in %post.  I dropped some
commands that started networking services and that seems to have allowed
install_root/dev to get unmounted.  install_root cannot be unmounted because the
livecd-creator process has loaded a shared library from the install_root:

[root@saga ~]# fuser -m /export/backup1/tmp/livecd-creator-cU3r1q/install_root
/export/backup1/tmp/livecd-creator-cU3r1q/install_root: 28172m
[root@saga ~]# ps -fe | grep 28172
root     12550 28172  7 17:30 pts/0    00:00:03 /sbin/resize2fs
/export/backup1/tmp/livecd-creator-cU3r1q/data/LiveOS/ext3fs.img 1120000
root     12623 12558  0 17:31 pts/1    00:00:00 grep 28172
root     28172 28104 21 16:28 pts/0    00:13:47 /usr/bin/python -tt
/usr/bin/livecd-creator -c rawhide-eeepc.cfg -t /export/backup1/tmp
[root@saga ~]# lsof -p 28172 | grep install
livecd-cr 28172 root  mem    REG        7,0    50900   468717
/export/backup1/tmp/livecd-creator-cU3r1q/install_root/lib/libnss_files-2.7.90.so
Comment 2 Jeremy Katz 2008-01-31 11:54:05 EST
Can you attach the config you're using (or send to me privately)?  In a quick
try here I'm not reproducing and before I spend a lot of time, it probably just
makes more sense to try with your config :)
Comment 3 Jeremy Katz 2008-01-31 13:56:13 EST
Okay, fixed the wiping out based on what you sent via mail (thanks).  Not seeing
what is keeping a file open, though.  Does that part happen reliably?
Comment 4 Orion Poplawski 2008-01-31 16:16:12 EST
I've yet to see it be able to successfully unmount install_root.

Any idea why livecd-creator is dynamically loading a shared library from the
install_root?   It appears to happen during the package install phase, during
one of these packages.

 Installing: gawk                         ################### [ 406/1156]
  Installing: module-init-tools            ################### [ 407/1156]
  Installing: shared-mime-info             ################### [
408/1156]warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root
warning: group polkituser does not exist - using root

  Installing: PolicyKit                    ################### [ 409/1156]
  Installing: libsndfile                   ################### [ 410/1156]
  Installing: dbus-x11                     ################### [ 411/1156]
  Installing: esound-libs                  ################### [ 412/1156]
  Installing: gd                           ################### [ 413/1156]
  Installing: logrotate                    ################### [ 414/1156]
  Installing: gzip                         ################### [ 415/1156]
  Installing: libxklavier                  ################### [ 416/1156]
  Installing: ppp                          ################### [ 417/1156]
  Installing: libsamplerate                ################### [ 418/1156]
  Installing: raptor                       ################### [ 419/1156]
  Installing: libuser                      ################### [ 420/1156]
  Installing: hdf5                         ################### [ 421/1156]
  Installing: libgssglue                   ################### [ 422/1156]
  Installing: Xaw3d                        ################### [ 423/1156]
  Installing: libXpm-devel                 ################### [ 424/1156]
  Installing: passwd                       ################### [ 425/1156]
  Installing: pulseaudio-core-libs         ################### [ 426/1156]
  Installing: rsyslog                      ################### [ 427/1156]
  Installing: vixie-cron                   ################### [ 428/1156]
  Installing: ConsoleKit                   ################### [ 429/1156]
  Installing: lvm2                         ################### [ 430/1156]
  Installing: ConsoleKit-libs              ################### [
431/1156]warning: group t
ty does not exist - using root

  Installing: util-linux-ng                ################### [ 432/1156]
  Installing: usermode                     ################### [ 433/1156]
  Installing: apr-util                     ################### [
434/1156]warning: group t

  Installing: sysvinit                     ################### [ 435/1156]
  Installing: xorg-x11-utils               ################### [ 436/1156]
  Installing: ruby-libs                    ################### [ 437/1156]
  Installing: ruby                         ################### [ 438/1156]
  Installing: mysql-libs                   ################### [ 439/1156]
  Installing: postgresql-libs              ################### [ 440/1156]
  Installing: genisoimage                  ################### [ 441/1156]
  Installing: pulseaudio-libs-glib2        ################### [ 442/1156]
  Installing: libIDL                       ################### [ 443/1156]
  Installing: ORBit2                       ################### [ 444/1156]
  Installing: GConf2                       ################### [ 445/1156]
  Installing: libbonobo                    ################### [ 446/1156]
  Installing: libgsf                       ################### [ 447/1156]
  Installing: texinfo-tex                  ################### [ 448/1156]
  Installing: hwdata                       ################### [ 449/1156]
  Installing: xdg-utils                    ################### [ 450/1156]
  Installing: hicolor-icon-theme           ################### [ 451/1156]
  Installing: redhat-menus                 ################### [ 452/1156]
  Installing: urw-fonts                    ################### [ 453/1156]
  Installing: fontconfig-devel             ################### [ 454/1156]
  Installing: ghostscript-fonts            ################### [ 455/1156]
  Installing: fedora-logos                 ################### [ 456/1156]
  Installing: autoconf                     ################### [ 457/1156]
  Installing: libXfixes-devel              ################### [ 458/1156]
  Installing: fedora-usermgmt-core         ################### [ 459/1156]
  Installing: automake                     ################### [ 460/1156]
Comment 5 Jeremy Katz 2008-01-31 16:24:37 EST
The only things I can think of would be an app starting/restarting in the %post
of a package.  But I haven't noticed that building things here.  I'll try with
your config now that my good build box is unhosed (ahhh, the fun of a rawhide
update after 4 weeks of not being in the office)
Comment 6 Jeremy Katz 2008-02-01 10:50:22 EST
Blah, and it of course unmounted fine for me.   Need to set up an F8 box for
some stuff today I think...
Comment 7 Orion Poplawski 2008-02-19 22:43:34 EST
Some more problems caused by not being able to unmount the install_root - looks
like bits aren't flushed to disk - I get fsck errors later in the creation phase:

umount: /export/backup1/tmp/imgcreate-VdChjT/install_root: device is busy
umount: /export/backup1/tmp/imgcreate-VdChjT/install_root: device is busy
e2fsck 1.40.4 (31-Dec-2007)
Pass 1: Checking inodes, blocks, and sizes
Extended attribute block 459261 has reference count 691, should be 609.  Fix? yes

Extended attribute block 459266 has reference count 71, should be 70.  Fix? yes

Extended attribute block 1114616 has reference count 45, should be 44.  Fix? yes

Pass 2: Checking directory structure
Entry 'nfs' in / (2) has deleted/unused inode 858.  Clear? yes

Entry 'scratch' in / (2) has deleted/unused inode 859.  Clear? yes

Entry 'local' in /opt (16001) has deleted/unused inode 18291.  Clear? yes

Entry 'home' in /export (32001) has deleted/unused inode 32971.  Clear? yes

Entry 'local' in /export (32001) has deleted/unused inode 32972.  Clear? yes

Entry 'tmp' in /export (32001) has deleted/unused inode 32974.  Clear? yes

Entry 'K73messagebus' in /etc/rc.d/rc0.d (224049) has deleted/unused inode
231600.  Clear? yes
.....

and so on.  Needles to say the image isn't usable.  Would using umount -l as
mentioned on the list help this?
Comment 8 Orion Poplawski 2008-02-19 23:25:40 EST
Okay, it started watching the install process and kept an eye on the files
livecd-creator has opened at the same time.  nss_files.so gets loaded while
install libutempter, which has the following script:

rpm -qp --scripts development/i386/os/Packages/libutempter-1.1.5-1.fc9.i386.rpm 
preinstall scriptlet (using /bin/sh):
{
    /usr/sbin/groupadd -g 22 -r -f utmp || :
    /usr/sbin/groupadd -g 35 -r -f utempter || :
}

So it looks like running groupadd (and probably useradd later in other scripts)
is the culprit.  No idea why the parent process would inherit this though, or
why it would stay loaded afterwards.
Comment 9 Jeremy Katz 2008-04-14 21:59:46 EDT
This will be fixed in Fedora 9 as we don't bind mount /dev anymore
Comment 10 Alan Pevec 2008-07-25 00:07:15 EDT
yes, $summary is fixed
but I got one fully updated F-9 box where I observe the same issues described in
comment 7
and it's again libnss keeping the mount busy:
livecd-cr 20160      root  mem       REG                7,0    57808      28547
/var/tmp/imgcreate-nv5xbf/install_root/lib64/libnss_files-2.8.so

On other also fully updated F9 box, all is fine.

On the problematic box one previous livecd session was interrupted, but machine
was rebooted and /var/tmp/ cleaned up in the meantime.

Any ideas are much appreciated...

Note You need to log in before you can comment on or make changes to this bug.