Bug 2054152

I wonder if this is actually an RFE - not sure if we currently support it. For direct kernel boot, we'd need to make sure that at least the initramfs has the virtiofs driver - can you check?

(In reply to Klaus Heinrich Kiwi from comment #1)
> I wonder if this is actually an RFE - not sure if we currently support it.
> For direct kernel boot, we'd need to make sure that at least the initramfs
> has the virtiofs driver - can you check?

Yes, I checked the initramfs that it has virtiofs.ko.xz.
# lsinitrd initramfs-5.14.0-55.el9.x86_64.img |grep virtiofs
-rw-r--r--   1 root     root        13496 Aug 24 10:27 usr/lib/modules/5.14.0-55.el9.x86_64/kernel/fs/fuse/virtiofs.ko.xz

(In reply to xiagao from comment #2)
> (In reply to Klaus Heinrich Kiwi from comment #1)
> > I wonder if this is actually an RFE - not sure if we currently support it.
> > For direct kernel boot, we'd need to make sure that at least the initramfs
> > has the virtiofs driver - can you check?
> 
> Yes, I checked the initramfs that it has virtiofs.ko.xz.
> # lsinitrd initramfs-5.14.0-55.el9.x86_64.img |grep virtiofs
> -rw-r--r--   1 root     root        13496 Aug 24 10:27
> usr/lib/modules/5.14.0-55.el9.x86_64/kernel/fs/fuse/virtiofs.ko.xz

Thanks Xiaogao. Have you tested this exact same scenario in RHEL 8.6? Can you share the results?

 -Klaus

@gmaglione have you also had a chance to take a look on this one?

The original documentation (https://virtio-fs.gitlab.io/howto-boot.html)
assumes a monolithic kernel and the use of an iniramfs requires extra steps.
Even if the virtiofs driver is present, dracut doesn't know what to do with it. 
So, you get the "dracut: FATAL: Don't know how to handle 'root=tag'" error message.


We need to do the following steps:
1. Create a fedora root fs:

# ./create_virtiofs_root.sh virtio-fs-root

(the 'create_virtiofs_root.sh' is at the end)

This will install a fedora 35, create a virtiofs dracut module (it's a modification of the 9p module)
add a user without password and rebuild the initramfs.
Since this requires to rebuild the initramfs, I'm not sure if it will work with a RHEL kernel in a fedora
installation.


2. start virtiofsd:

# /usr/libexec/virtiofsd --socket-path=/tmp/sock1 -o source=$PWD/virtio-fs-root/ -o cache=none


3. boot up vm from virtiofs, tag=host and add "root=virtiofs:host" to kernel.
(you can use any tag, even tag=/dev/root and add "root=virtiofs:/dev/root")

# qemu-kvm \
......
-chardev socket,id=char0,path=/tmp/sock1 \
-device vhost-user-fs-pci,chardev=char0,tag=host,bus=pci.8,queue-size=1024,bootindex=1 \
-object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on \
-numa node,memdev=mem \
-kernel '$PWD/vmlinuz-5.14.0-55.el9.x86_64'  \
-initrd '$PWD/initramfs-5.14.0-55.kpq0.el9.x86_64.img'  \
-append "root=virtiofs:host"


------------  BEGIN: create_virtiofs_root.sh ------------
#!/bin/bash
set -u

# Fedora release version
release_version=35

if [ $# -lt 1 ]; then
	echo "Usage: $0 <virtiofs_root_dir>"
	exit 1
fi

virtiofs_root=$(realpath ${1})

# Install the base system
dnf --installroot=${virtiofs_root} --releasever=${release_version} --assumeyes install system-release vim-minimal systemd passwd dnf rootfiles sudo kernel
cd ${virtiofs_root}

# Add dracut module
mkdir -p usr/lib/dracut/modules.d/95virtiofs

cat > usr/lib/dracut/modules.d/95virtiofs/module-setup.sh  <<'EOF'
#!/usr/bin/bash

# called by dracut
check() {
    [[ $hostonly ]] || [[ $mount_needs ]] && {
        for fs in "${host_fs_types[@]}"; do
            [[ $fs == "virtiofs" ]] && return 0
        done
        return 255
    }

    is_qemu_virtualized && return 0

    return 255
}

# called by dracut
depends() {
    return 0
}

# called by dracut
installkernel() {
    instmods virtiofs
}

# called by dracut
install() {
    inst_hook cmdline 95 "$moddir/parse-virtiofs.sh"
    inst_hook pre-mount 99 "$moddir/mount-virtiofs.sh"
}
EOF

cat > usr/lib/dracut/modules.d/95virtiofs/parse-virtiofs.sh  <<'EOF'
#!/usr/bin/sh

if [ "${root%%:*}" = "virtiofs" ]; then
    modprobe virtiofs

    # shellcheck disable=SC2034
    rootok=1
fi
EOF

cat > usr/lib/dracut/modules.d/95virtiofs/mount-virtiofs.sh  <<'EOF'
#!/usr/bin/sh

type getarg > /dev/null 2>&1 || . /lib/dracut-lib.sh

filter_rootopts() {
    rootopts=$1
    # strip ro and rw options
    local OLDIFS="$IFS"
    IFS=,
    # shellcheck disable=SC2086
    set -- $rootopts
    IFS="$OLDIFS"
    local v
    while [ $# -gt 0 ]; do
        case $1 in
            rw | ro) ;;
            defaults) ;;
            *)
                v="$v,${1}"
                ;;
        esac
        shift
    done
    rootopts=${v#,}
    echo "$rootopts"
}

mount_root() {
    rootfs="virtiofs"
    rflags="rw"

    modprobe virtiofs

    mount -t ${rootfs} -o "$rflags",ro "${root#virtiofs:}" "$NEWROOT"

    rootopts=
    if getargbool 1 rd.fstab -n rd_NO_FSTAB \
        && ! getarg rootflags \
        && [ -f "$NEWROOT/etc/fstab" ] \
        && ! [ -L "$NEWROOT/etc/fstab" ]; then
        # if $NEWROOT/etc/fstab contains special mount options for
        # the root filesystem,
        # remount it with the proper options
        rootopts="defaults"
        while read -r dev mp _ opts rest || [ -n "$dev" ]; do
            # skip comments
            [ "${dev%%#*}" != "$dev" ] && continue

            if [ "$mp" = "/" ]; then
                rootopts=$opts
                break
            fi
        done < "$NEWROOT/etc/fstab"

        rootopts=$(filter_rootopts "$rootopts")
    fi

    # we want rootflags (rflags) to take precedence so prepend rootopts to
    # them; rflags is guaranteed to not be empty
    rflags="${rootopts:+${rootopts},}${rflags}"

    umount "$NEWROOT"

    info "Remounting ${root#virtiofs:} with -o ${rflags}"
    mount -t ${rootfs} -o "$rflags" "${root#virtiofs:}" "$NEWROOT" 2>&1 | vinfo

    [ -f "$NEWROOT"/forcefsck ] && rm -f -- "$NEWROOT"/forcefsck 2> /dev/null
    [ -f "$NEWROOT"/.autofsck ] && rm -f -- "$NEWROOT"/.autofsck 2> /dev/null
}

if [ -n "$root" -a -z "${root%%virtiofs:*}" ]; then
    mount_root
fi
:
EOF

chmod +x usr/lib/dracut/modules.d/95virtiofs/*

# Add a user with no password and sudo,
# and rebuild the initramfs adding virtiofs filesystem and module
cat > prepare.sh  <<'EOF'
#!/bin/bash

useradd -m -p '' -G wheel user

cd /boot
kernel_file=$(basename vmlinuz-*)
kversion=${kernel_file##vmlinuz-}

initramfs_img=initramfs-${kversion}.img
mv ${initramfs_img} ${initramfs_img}.bak

dracut --kver ${kversion} --add virtiofs --filesystems virtiofs --force
EOF

chmod +x prepare.sh


# This is required by dracut
mount --bind /sys 	sys
mount --bind /proc 	proc
mount --bind /run 	run
mount --bind /dev 	dev
mount --bind /dev/pts 	dev/pts

chroot . /prepare.sh

umount dev/pts
umount dev
umount run
umount proc
umount sys


------------  END: create_virtiofs_root.sh ------------

Since you want to use the RHEL9 kernel, you could modify the RHEL9 initramfs "by hand"

1. Extract the initramfs

# mkdir initramfs

# cd initramfs

# zcat path/to/initramfs-5.14.0-55.el9.x86_64.img | cpio -i


2. copy/edit dracut files

# cp path/to/parse-virtiofs.sh usr/lib/dracut/hooks/cmdline/95-parse-virtiofs.sh

# cp path/to/mount-virtiofs.sh usr/lib/dracut/hooks/pre-mount/99-mount-virtiofs.sh

# echo virtiofs >> usr/lib/dracut/modules.txt
(be careful with the '>>')


3. rebuild the initramfs

# find . | cpio -o -H newc | gzip > ../initramfs-5.14.0-55.el9.x86_64.virtiofs.img

Hi German,
Very detailed notes,thanks. And I tried to unzip initramfs but failed to get "usr/lib/dracut/hooks/cmdline/95-parse-virtiofs.sh" directory.

# lsinitrd initramfs-5.14.0-66.el9.x86_64.img |grep virtiofs
-rw-r--r--   1 root     root        13472 Aug 24  2021 usr/lib/modules/5.14.0-66.el9.x86_64/kernel/fs/fuse/virtiofs.ko.xz


# zcat ../initramfs-5.14.0-66.el9.x86_64.img |cpio -i
gzip: ../initramfs-5.14.0-66.el9.x86_64.img: not in gzip format
cpio: premature end of archive


# cpio -id < ../initramfs-5.14.0-66.el9.x86_64.img 
11401 blocks

# tree
.
├── early_cpio
└── kernel
    └── x86
        └── microcode
            ├── AuthenticAMD.bin
            └── GenuineIntel.bin

I forgot to mention that both scripts 'parse-virtiofs.sh' and 'mount-virtiofs.sh' are part of the script in comment 7

The RHEL initramfs it's an image with the microde prepended, you can check this by:

# lsinitrd initramfs-5.14.0-66.el9.x86_64.img
========================================================================
Early CPIO image
========================================================================
drwxr-xr-x   3 root     root            0 Aug 24  2021 .
-rw-r--r--   1 root     root            2 Aug 24  2021 early_cpio
...
========================================================================
Version: dracut-055-10.git20210824.el9

Arguments:  -f

dracut modules:
bash
...
========================================================================
drwxr-xr-x  12 root     root            0 Aug 24  2021 .
crw-r--r--   1 root     root       5,   1 Aug 24  2021 dev/console
...
drwxr-xr-x   2 root     root            0 Aug 24  2021 var/tmp
========================================================================

so, to modify it we need to:

1. Extract the images

Get the early microcode initramfs (we will not change this image)
We have two options:
First we need to find how many blocks the early microcode occupies
# cpio -t < /path/to/initramfs-5.14.0-66.el9.x86_64.img 
.
early_cpio
kernel
...
11401 blocks
# dd if=/path/to/initramfs-5.14.0-66.el9.x86_64.img  of=early.img count=11401

Sometimes cpio reports the wrong number of blocks, so we can do this instead
# mkdir early_init
# cd early_init
# lsinitrd --unpackearly  /path/to/initramfs-5.14.0-66.el9.x86_64.img
# cpio -o -H newc > ../early.img
The "early.img" it's not a compressed image


2 Extract the initramfs

# mkdir initramfs
# cd initramfs
# lsinitrd --unpack /path/to/initramfs-5.14.0-66.el9.x86_64.img

Or if lsinitrd it's not available
# dd if=/path/to/initramfs-5.14.0-66.el9.x86_64.img  of=initramfs.img skip=11401
# mkdir initramfs
# cd initramfs
# zcat ../initramfs.img | cpio -i


 
3 Copy/edit dracut files

# cp /path/to/parse-virtiofs.sh usr/lib/dracut/hooks/cmdline/95-parse-virtiofs.sh
# cp /path/to/mount-virtiofs.sh usr/lib/dracut/hooks/pre-mount/99-mount-virtiofs.sh
# echo virtiofs >> usr/lib/dracut/modules.txt
(be careful with the '>>')


3. Rebuild the initramfs
# find . | cpio -o -H newc | gzip > ../initramfs.virtiofs.img
# cd ..
# cat early.img initramfs.virtiofs.img > initramfs-5.14.0-66.el9.virtiofs.x86_64.img

If you have a RHEL distribution, you can rebuild the initramfs as follows:

Add the dracut module
# mkdir -p /usr/lib/dracut/modules.d/95virtiofs
# cp module-setup.sh /usr/lib/dracut/modules.d/95virtiofs
# cp parse-virtiofs.sh /usr/lib/dracut/modules.d/95virtiofs
# cp mount-virtiofs.sh /usr/lib/dracut/modules.d/95virtiofs
# chmod +x /usr/lib/dracut/modules.d/95virtiofs/*

(module-setup.sh, parse-virtiofs.sh and mount-virtiofs.sh are included in the script in the Comment 7)

Create the initramfs
# dracut --kver ${kernel_version}  --early-microcode --add virtiofs --filesystems virtiofs

Probably it's a good idea to include all the modules and filesystems listed in:
# lsinitrd -m initramfs-${kernel_version}.img

If you need to this many times, it's better to set up a dracut.conf
Adding all modules and filesystems listed by lsinitrd

cat > /etc/dracut.conf.d/virtiofs.conf  <<EOF 
early_microcode="yes"
add_dracutmodules+=" virtiofs "
filesystems+=" virtiofs "

EOF

And just calling:
# dracut --kver ${kernel_version}

Hi German,
Sorry for the late reply, I just return to this issue.
And I tried it just now with comment 16, it worked well and I updated to our test plan.
Thank you very much for all the detailed methods above.

Hi Klaus,
I tested the method German provided, it works.So I think we can close this bug. How do you think?

Thanks,
Xiaoling

(In reply to xiagao from comment #18)
> Hi German,
> Sorry for the late reply, I just return to this issue.
> And I tried it just now with comment 16, it worked well and I updated to our
> test plan.
> Thank you very much for all the detailed methods above.
> 
> Hi Klaus,
> I tested the method German provided, it works.So I think we can close this
> bug. How do you think?
> 

I think we need to create a new BZ with an RFE against dracut (or the default configuration of the initramfs in RHEL guest images), and I think German will take this step.

For this bug, as it was a TestOnly demonstrated to work, I think we should close it and reference to it on the bug above.

Thanks

German, we probably need a dracut bug also to get virtiofs related changes in dracut.

I have sent a PR to add virtiofs as a dracut module.
https://github.com/dracutdevs/dracut/pull/1784

Hi German, do we want to have it on RHE9.1.0? If yes, could you help add 'Internal Target Release' flag? Thanks.

I've set ITR 9.1, because BZ 1967958 did it, also they have set ITM to 26

Hi all,
The test still failed with dracut-057-13.git20220816.el9(fix in Bug 1967958) and I checked that virtiofs module was already added to dracut in my host, but I didn't find virtiofs.ko.xz in initramfs image.

# ll /usr/lib/dracut/modules.d/95virtiofs
total 12
-rwxr-xr-x. 1 root root 531 Jun 19 18:35 module-setup.sh
-rwxr-xr-x. 1 root root 637 Jun 19 18:35 mount-virtiofs.sh
-rwxr-xr-x. 1 root root 208 Jun 19 18:35 parse-virtiofs.sh

# rpm -qa |grep dracut
dracut-057-13.git20220816.el9.x86_64

[root@dell-per440-06 modules.d]# lsinitrd /boot/initramfs-5.14.0-145.el9.x86_64.img |grep virtiofs
[root@dell-per440-06 modules.d]#

Steps:
1. # mkdir $PWD/virtio-fs-root
2. # dnf --installroot=$PWD/virtio-fs-root --releasever=9 install system-release vim-minimal systemd passwd dnf rootfiles

3. start virtiofsd
# /usr/libexec/virtiofsd --socket-path=/tmp/sock1 -o source=$PWD/virtio-fs-root/ -o cache=none

4. copy a RHEL9 vm kernel and initramfs file to current dir

5. boot up vm from virtiofs, tag=myfs1 and add "root=myfs1 rootfstype=virtiofs" to kernel.
# qemu-kvm \
......
        -object memory-backend-file,id=mem,size=8G,mem-path=/dev/shm,share=on \
        -numa node,memdev=mem \
        -chardev socket,id=char0,path=/tmp/sock1 \
        -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs1 \
        -kernel '$PWD/vmlinuz-5.14.0-145.el9.x86_64'  \
        -initrd '$PWD/initramfs-5.14.0-145.el9.x86_64.img'  \
        -append "root=myfs1 rootfstype=virtiofs" \

Actual results:
dracut: Don't know how to handle 'root=myfs1'
dracut: Refusing to continue

......
reboot: System halted

Hi German, Could you help to look at comment 29? Thanks.

(In reply to xiagao from comment #30)
> Hi German, Could you help to look at comment 29? Thanks.

Hi

The RHEL 9.1 stock initramfs is not created with the new module, I'll open a BZ.

So in the meantime, you can do what I posted in comment 16 from "Create the initramfs",
since dracut already includes the virtiofs module, all you need to do is:

# dracut initramfs-virtiofs.img  --early-microcode --add virtiofs --filesystems virtiofs

Just an extra bit of info.

The virtiofs module is automatically included, unless dracut is called with --hostonly on a host that is not a guest or does not use virtiofs. You can see this in the chec() function inside module-setup.sh
(returning 0 means being included):

check() {
    [[ $hostonly ]] || [[ $mount_needs ]] && {
        is_qemu_virtualized && return 0

        for fs in "${host_fs_types[@]}"; do
            [[ $fs == "virtiofs" ]] && return 0
        done
        return 255
    }

    return 0
}


By default, dracut is called with --hostonly If you just run "dracut image.img"

(In reply to xiagao from comment #29)
> Hi all,
> The test still failed with dracut-057-13.git20220816.el9(fix in Bug 1967958)
> and I checked that virtiofs module was already added to dracut in my host,
> but I didn't find virtiofs.ko.xz in initramfs image.
> 
> # ll /usr/lib/dracut/modules.d/95virtiofs
> total 12
> -rwxr-xr-x. 1 root root 531 Jun 19 18:35 module-setup.sh
> -rwxr-xr-x. 1 root root 637 Jun 19 18:35 mount-virtiofs.sh
> -rwxr-xr-x. 1 root root 208 Jun 19 18:35 parse-virtiofs.sh
> 
> # rpm -qa |grep dracut
> dracut-057-13.git20220816.el9.x86_64
> 
> [root@dell-per440-06 modules.d]# lsinitrd
> /boot/initramfs-5.14.0-145.el9.x86_64.img |grep virtiofs
> [root@dell-per440-06 modules.d]#
> 
> Steps:
> 1. # mkdir $PWD/virtio-fs-root
> 2. # dnf --installroot=$PWD/virtio-fs-root --releasever=9 install
> system-release vim-minimal systemd passwd dnf rootfiles
> 
> 3. start virtiofsd
> # /usr/libexec/virtiofsd --socket-path=/tmp/sock1 -o
> source=$PWD/virtio-fs-root/ -o cache=none
> 
> 4. copy a RHEL9 vm kernel and initramfs file to current dir

I am wondering what's the goal here? So you are taking an initramfs
(generated for rhel9 system) and trying to use same initramfs to
boot a VM with different root filesystem.

I don't think that will work. Typically initramfs is generated
after installation and it contains modules which are needed for
booting that system. Now if your host has "xfs" as root
filesystem, you can't use that initramfs to boot into another
type of root filesystem. For example, that initramfs might
not have "ext4" modules and you might not be able to boot
into "ext4" rootfs using that initramfs.

So my understanding is that you will have to re-generate initramfs
again with virtiofs module and use that initramfs to boot into
a VM using virtiofs as rootfs.

IOW, my understanding is that initramfs is supposed to be minimal and just contains enough modules to be able to boot target system. If you take that initramfs to some other system with different storage and filesystem module stack needed, it might not work. So if you want to boot a VM with virtiofs as rootfs, then you will need to use an initramfs which container "virtiofs" module in it and RHEL9 initramfs might not have it by default.

(In reply to Vivek Goyal from comment #34)
> IOW, my understanding is that initramfs is supposed to be minimal and just
> contains enough modules to be able to boot target system. If you take that
> initramfs to some other system with different storage and filesystem module
> stack needed, it might not work. So if you want to boot a VM with virtiofs
> as rootfs, then you will need to use an initramfs which container "virtiofs"
> module in it and RHEL9 initramfs might not have it by default.

You're right. The best way is to create a new initramfs including virtiofs modules, we are using initramfs on the host is just to make things/test easier(don't need to create a new one,to be honest I'm not familiar with creating a new initramfs). 
Thanks for your update.

(In reply to German Maglione from comment #32)
> Just an extra bit of info.
> 
> The virtiofs module is automatically included, unless dracut is called with
> --hostonly on a host that is not a guest or does not use virtiofs. You can
> see this in the chec() function inside module-setup.sh
> (returning 0 means being included):
> 
> check() {
>     [[ $hostonly ]] || [[ $mount_needs ]] && {
>         is_qemu_virtualized && return 0
> 
>         for fs in "${host_fs_types[@]}"; do
>             [[ $fs == "virtiofs" ]] && return 0
>         done
>         return 255
>     }
> 
>     return 0
> }
> 
> 
> By default, dracut is called with --hostonly If you just run "dracut
> image.img"

Hi German, it works. Thank you very much.
The bug status is still NEW, can we close it now?  As D/T phase will be end of ITM 26 (2022/8/29), all bzs fix need exception+/blocker+ after ITM26.

Thanks,
Xiaoling

(In reply to xiagao from comment #36)
> Hi German, it works. Thank you very much.
> The bug status is still NEW, can we close it now?  As D/T phase will be end
> of ITM 26 (2022/8/29), all bzs fix need exception+/blocker+ after ITM26.


I think yes we can close it, I consider this fixed since the current behavior is expected.

Thanks,

I will close it as current release, as I see the fix is already in dracut.
# rpm -qa |grep dracut
dracut-057-13.git20220816.el9.x86_64
kernel 5.14.0-145.el9.x86_64