Bug 503169 - libguestfs /dev is too sparse for kernel installation/upgrade
libguestfs /dev is too sparse for kernel installation/upgrade
Status: CLOSED UPSTREAM
Product: Virtualization Tools
Classification: Community
Component: libguestfs (Show other bugs)
unspecified
All Linux
low Severity medium
: ---
: ---
Assigned To: Richard W.M. Jones
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-29 07:28 EDT by Charles Duffy
Modified: 2010-03-27 06:15 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-03-27 06:15:17 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
LIBGUESTFS_DEBUG log demonstrating sparse /dev (13.02 KB, text/plain)
2009-06-01 14:35 EDT, Charles Duffy
no flags Details
mount /dev via tmpfs before MAKEDEV invocation (454 bytes, patch)
2009-06-03 19:34 EDT, Charles Duffy
no flags Details | Diff
Use --rbind rather than --bind for bind mounting /dev (to get /dev/pts) (874 bytes, patch)
2009-06-03 19:35 EDT, Charles Duffy
no flags Details | Diff

  None (edit)
Description Charles Duffy 2009-05-29 07:28:26 EDT
Richard W.M. Jones wrote:
> On Fri, May 29, 2009 at 05:50:47AM -0500, Charles Duffy wrote:
>> I'm trying to upgrade kernel packages via libguestfs -- but grubby and  
>> mkinitrd don't deal well at all with having a less-than-fully-populated  
>> /dev, either failing to operate at all or producing an initrd with LVM  
>> support omitted.
>>
>> Before I spend too much more time on this one -- any hints or recipes?
>
> Quite probably we need to add more /dev devices.
>
> For reasons unknown, the base filesystem RPM in Red Hat distributions
> doesn't set up any device nodes.  'mock' has to do this itself.  We do
> the same thing (copying code from 'mock') in make-initramfs.
>
> Can you identify which ones are needed?  Try adding them to the
> make-initramfs '/init' script.
>
> As ever, once you get more details open a BZ so we can track this
> issue.

The following commands, run in the order given, appear to create all entries in /dev necessary for make-initramfs and grubby to work on RHEL5:

MAKEDEV -x /dev/mapper /dev/null $(cat /proc/partitions | gawk '{print "/dev/" $4}' | egrep -v '/dev/(dm-|name|$)')
lvm pvscan
lvm vgscan
lvm vgchange -a y
Comment 1 Richard W.M. Jones 2009-05-29 07:58:09 EDT
(In reply to comment #0)
> The following commands, run in the order given, appear to create all entries in
> /dev necessary for make-initramfs and grubby to work on RHEL5:
> 
> MAKEDEV -x /dev/mapper /dev/null $(cat /proc/partitions | gawk '{print "/dev/"
> $4}' | egrep -v '/dev/(dm-|name|$)')
> lvm pvscan
> lvm vgscan
> lvm vgchange -a y  

These commands are run after the /init script, using
guestfs_command or similar?

What actual device nodes get added by the above?
eg. try 'll /../dev' before and after and look at the
diff between the two.

AFAICT /dev/null should exist - not much would work
without it.  /dev/mapper exists.

><fs> ll /../dev/mapper
total 0
drwxrwxr-x 2 root root 0 May 29 07:03 .
drwxr-xr-x 5 root root 0 May 29 07:57 ..

><fs> ll /../dev/null
crw-rw-rw- 1 root root 1, 3 May 29 07:08 /sysroot/../dev/null

I need to know what needs to be run in the actual /init
script to fix things.
Comment 2 Charles Duffy 2009-05-29 16:16:21 EDT
The issue is not with /dev, but with /sysroot/dev. By default, /sysroot/dev has only the LVM-created devices, mapper, and null:

VolGroup00
mapper
null

After running the previously-given commands via a script invoked from guestfs_command:

VolGroup00
mapper
null
hda
hda1
hda2
hdb

I've tried "mount-options bind /dev /dev" to pull in the much more thoroughly populated contents of /../dev to /sysroot/dev, but the command fails as given.
Comment 3 Richard W.M. Jones 2009-06-01 04:58:49 EDT
I think you need to run this with LIBGUESTFS_DEBUG=1
to see what's really happening.  When we run commands
in the guest, we first bind-mount /dev into /sysroot/dev,
/proc into /sysroot/proc and /sys into /sysroot/sys.

See: http://git.et.redhat.com/?p=libguestfs.git;a=blob;f=daemon/command.c;h=1a50264b3c4c100b6d0c6f467ccc3e06a4c5faa1;hb=HEAD#l48

Now it may be for some reason this isn't happening
or is not doing the right thing.  The only way to find
out for sure is to look for messages printed out
by the guest.
Comment 4 Charles Duffy 2009-06-01 14:35:36 EDT
Created attachment 346117 [details]
LIBGUESTFS_DEBUG log demonstrating sparse /dev

In the given log (with LIBGUESTFS_DEBUG=1), we successfully mount /dev/hda1, but /dev contains no such device.
Comment 5 Richard W.M. Jones 2009-06-01 15:12:02 EDT
Thanks for doing that.  The output is seriously odd:

VolGroup00
mapper
null

I actually see almost the same thing.  For example with
a Fedora 10 guest:

><fs> command "ls -l /dev"
total 8
drwxr-xr-x 2 root root 4096 May 10 21:36 VolGroup00
drwxr-xr-x 2 root root 4096 May 10 21:36 mapper

This is a very odd one because it looks as if the
bind-mount is somehow only copying directories, not
device nodes.  It might be something to do with the
permissions on /dev, or something about bind-mounts
in the kernel, or something else.

--

By the way, in the test you try to run the command
"ls /../dev", but because the command runs inside the
chroot, that's exactly the same as running "ls /dev".
The same goes for the direct ls /../dev command which
already runs in a chroot[1].  The only command which doesn't
run in a chroot is ll[2].  It's very helpful to look at
the source to see the precise semantics.

[1] http://git.et.redhat.com/?p=libguestfs.git;a=blob;f=daemon/ls.c;h=261bc96481f2e97b7ec93a31ac44590af3f12746;hb=HEAD#l32

[2] http://git.et.redhat.com/?p=libguestfs.git;a=blob;f=daemon/ls.c;h=261bc96481f2e97b7ec93a31ac44590af3f12746;hb=HEAD#l87
Comment 6 Charles Duffy 2009-06-03 19:05:07 EDT
I should have recognized this one, having run into it in the past:

initramfs can't be bind mounted.

If the sysroot is modified to put /dev on a different filesystem (such as tmpfs), that should resolve the issue.
Comment 7 Charles Duffy 2009-06-03 19:34:14 EDT
Created attachment 346475 [details]
mount /dev via tmpfs before MAKEDEV invocation
Comment 8 Charles Duffy 2009-06-03 19:35:08 EDT
Created attachment 346476 [details]
Use --rbind rather than --bind for bind mounting /dev (to get /dev/pts)
Comment 10 Richard W.M. Jones 2009-06-04 05:11:48 EDT
This causes a regression.  There are existing /dev
nodes in the initramfs which are hidden by the tmpfs.
The existing nodes are:

$ ls -l initramfs/dev/
total 12
-rw------- 1 rjones rjones    0 2009-06-04 09:10 console
-rw-rw-rw- 1 rjones rjones    0 2009-06-04 09:10 full
drwxrwxr-x 2 rjones rjones 4096 2009-06-04 09:10 mapper
-rw-rw-rw- 1 rjones rjones    0 2009-06-04 09:11 null
-rw-rw-rw- 1 rjones rjones    0 2009-06-04 09:10 ptmx
drwxrwxr-x 2 rjones rjones 4096 2009-06-04 09:10 pts
-rw-rw-rw- 1 rjones rjones    0 2009-06-04 09:10 random
drwxrwxr-x 2 rjones rjones 4096 2009-06-04 09:10 shm
lrwxrwxrwx 1 rjones rjones   15 2009-06-04 09:10 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 rjones rjones   15 2009-06-04 09:10 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 rjones rjones   15 2009-06-04 09:10 stdout -> /proc/self/fd/1
-rw-rw-rw- 1 rjones rjones    0 2009-06-04 09:10 tty
-rw-r--r-- 1 rjones rjones    0 2009-06-04 09:10 urandom
-rw-rw-rw- 1 rjones rjones    0 2009-06-04 09:10 zero

Moving back to ASSIGNED.
Comment 11 Richard W.M. Jones 2009-06-04 05:59:27 EDT
I think this should fix the regression:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=a903368c6db67dc599e5cbefe9e41611fba81faf
Comment 12 Fedora Update System 2009-06-04 07:06:45 EDT
libguestfs-1.0.40-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.40-1.fc11
Comment 13 Richard W.M. Jones 2009-06-04 07:58:37 EDT
This doesn't fully fix the regression.

/sysroot/dev is not unmounted after the command has run,
consequently we get:

umount_all: umount: /sysroot/dev: umount: /sysroot/dev: device is busy.

Possibly --rbind cannot be reversed by a simple unmount
but you have to unmount all the subdevices too.

Setting bug back to ASSIGNED.
Comment 14 Richard W.M. Jones 2009-06-04 09:00:47 EDT
><fs> mounts
/dev/sda2
/dev/sda1
><fs> command "ls -l /"
total 73
drwxr-xr-x    2 root     root         4096 Feb 12 13:39 bin
drwxr-xr-x    4 root     root         1024 Feb 12 13:55 boot
drwxrwxrwt    5 root     root       173480 Jun  4 13:56 dev
drwxr-xr-x   64 root     root         4096 Feb 27 13:41 etc
drwxr-xr-x    3 root     root         4096 Feb 12 13:58 home
drwxr-xr-x    2 root     root         4096 Jan 24  2003 initrd
drwxr-xr-x   11 root     root         4096 Feb 12 13:49 lib
drwx------    2 root     root        16384 Feb 12 13:35 lost+found
drwxr-xr-x    2 root     root         4096 Apr 11  2007 misc
drwxr-xr-x    4 root     root         4096 Feb 12 17:04 mnt
drwxr-xr-x    2 root     root         4096 Jan 24  2003 opt
dr-xr-xr-x   43 root     root            0 Jun  4 13:56 proc
drwxr-x---    6 root     root         4096 Feb 12 17:29 root
drwxr-xr-x    2 root     root         8192 Feb 12 13:41 sbin
drwxrwxrwt   11 root     root         4096 Feb 27 13:41 tmp
drwxr-xr-x   15 root     root         4096 Feb 12 13:36 usr
drwxr-xr-x   22 root     root         4096 Feb 12 13:49 var

><fs> mounts
/dev/sda2
/dev/sda1
/dev

Notice the additional bogus "/dev" mount afterwards.
Comment 15 Richard W.M. Jones 2009-06-04 10:08:31 EDT
(In reply to comment #10)
> This causes a regression.  There are existing /dev
> nodes in the initramfs which are hidden by the tmpfs.

Added a regression test to catch this if it reoccurs
in future:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=96b9d82e2f4f24cdd41040f54a7597202c04cf7e

(In reply to comment #13)
> This doesn't fully fix the regression.
> 
> /sysroot/dev is not unmounted after the command has run,
> consequently we get:
> 
> umount_all: umount: /sysroot/dev: umount: /sysroot/dev: device is busy.

Fixed this and added a regression test for it:

http://git.et.redhat.com/?p=libguestfs.git;a=commitdiff;h=ab81d09a710242ab61001ef5e8ffd321725b47a7
Comment 16 Fedora Update System 2009-06-04 11:40:21 EDT
libguestfs-1.0.41-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.41-1.fc11
Comment 17 Fedora Update System 2009-06-23 05:15:11 EDT
libguestfs-1.0.51-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.51-1.fc11
Comment 18 Fedora Update System 2009-06-29 15:27:50 EDT
libguestfs-1.0.54-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.54-2.fc11
Comment 19 Fedora Update System 2009-07-02 19:26:37 EDT
libguestfs-1.0.55-1.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.55-1.fc11
Comment 20 Fedora Update System 2009-07-11 05:27:36 EDT
libguestfs-1.0.58-2.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libguestfs-1.0.58-2.fc11
Comment 21 Fedora Update System 2009-07-11 13:12:58 EDT
libguestfs-1.0.54-2.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 22 Fedora Update System 2009-07-19 06:28:15 EDT
libguestfs-1.0.58-2.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 23 Richard W.M. Jones 2010-03-27 06:15:17 EDT
Long fixed upstream.

Note You need to log in before you can comment on or make changes to this bug.