seems like a kernel module is missing.. just don't know which

Anyway way to find out?  Any way to get better debugging?

boot from installation medium:

# lsmod

boot from not working installation with "rd.break=initqueue"

# lsmod

There's no installation medium AFAIK, it's using kickstart, and the error
occurs after installation during the first boot.

I will try adding the second suggestion later.

(In reply to Richard W.M. Jones from comment #5)
> There's no installation medium AFAIK, it's using kickstart, and the error
> occurs after installation during the first boot.
> 
> I will try adding the second suggestion later.

then do lsmod in %post and parse the logs

Happens on Fedora 25 (ppc64 & ppc64le) too.  I'll see if I can collect
the lsmod data ...

This is for Fedora 25 ppc64le:

It's somewhat painful to get lsmod output from %post.

lsmod in the initramfs:

Module                  Size  Used by
virtio_pci             23047  0
virtio_ring            20275  1 virtio_pci
crc32c_vpmsum          10227  0
virtio                 11218  1 virtio_pci

virtio-scsi is required to mount the root filesystem.

Interestingly the kickstart was done using a virtio-blk device
so I suppose it's including that driver instead of virtio-scsi
in the initramfs.

I suppose we can rerun dracut in %post and add virtio-scsi drivers,
since ideally we'd want to make a disk image which could be booted
with either virtio-blk or virtio-scsi.

This works fine on non-POWER architectures.

Annoyingly, using dracut --add-drivers virtio-scsi in the %post script
did not help, so I guess it's some other thing.

FINALLY I managed to work around this by doing:

%post

pushd /etc/dracut.conf.d
echo 'add_drivers+="virtio-blk virtio-scsi"' > virt-builder-virtio-scsi.conf
popd

KERNEL_VERSION=$(rpm -q kernel --qf '%{version}-%{release}.%{arch}\n' | sort -V | tail -1)
dracut -f /boot/initramfs-$KERNEL_VERSION.img $KERNEL_VERSION

%end

However this behaviour still differs from both RHEL 7.2 on ppc64
and RHEL 7 on all other architectures.

device-mapper.ko is also missing.  It should be included because
the guest was built with LVM support using `autopart --type=lvm'.

Something is very broken with dracut, RHEL 7.3 & POWER.

The kickstart I'm using now is:

install
text
reboot
lang en_US.UTF-8
keyboard us
network --bootproto dhcp
rootpw builder
firewall --enabled --ssh
timezone --utc America/New_York
selinux --enforcing

bootloader --location=mbr --append="console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH"

zerombr
clearpart --all --initlabel
autopart --type=lvm

# Halt the system once configuration has finished.
poweroff

%packages
@core
%end

%post
# Enable virtio-scsi support.
pushd /etc/dracut.conf.d
echo 'add_drivers+="virtio-blk virtio-scsi"' > virt-builder-virtio-scsi.conf
popd
# To make dracut config changes permanent, we need to rerun dracut.
# Rerun dracut for the installed kernel (not the running kernel).
KERNEL_VERSION="$(rpm -q kernel --qf '%{version}-%{release}.%{arch}\n' | sort -V | tail -1)"
dracut -f /boot/initramfs-$KERNEL_VERSION.img $KERNEL_VERSION
%end

Sorry, ignore the previous comment.  dm_*.ko modules are present.
The boot problem is something else.

please add spaces:

echo 'add_drivers+=" virtio-blk virtio-scsi "' > virt-builder-virtio-scsi.conf

OK I finally got this to boot.  I had to add virtio-scsi modules
to dracut (on ppc64/ppc64le only), so the situation is as in comment 10.
Ignore comment 11.

(In reply to Harald Hoyer from comment #13)
> please add spaces:
> 
> echo 'add_drivers+=" virtio-blk virtio-scsi "' >
> virt-builder-virtio-scsi.conf

I've updated the KS.  However it worked even without the spaces.

hmm, /usr/lib/modules.d/90kernel-modules/module-setup.sh has

        # install virtual machine support
        instmods virtio virtio_blk virtio_ring virtio_pci virtio_scsi \
            "=drivers/pcmcia" =ide "=drivers/usb/storage"


So, if the drivers are loaded on kickstart/installation time, they should be included in the final initramfs.

If you don't want the "hostonly" feature:


%packages
@core
dracut-config-generic
%end

which then should generate an initramfs with _all_ the kernel modules (and not just those, which were loaded on installation time)

I think what's needed first is a very clear and simple reproducer of
the original bug.  This is the most minimal reproducer I could create:

(1) Create a file 'rhel.ks' containing:

install
text
reboot
lang en_US.UTF-8
keyboard us
network --bootproto dhcp
rootpw builder
firewall --enabled --ssh
timezone --utc America/New_York
selinux --enforcing

bootloader --location=mbr --append="console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH"

zerombr
clearpart --all --initlabel
autopart --type=lvm

# Halt the system once configuration has finished.
poweroff

%packages
@core
%end


(2) Run the following virt-install command (as non-root if you wish):

virt-install --name=tmp \
    --ram=4096 \
    --arch=ppc64le \
    --machine=pseries \
    --cpu=POWER8 \
    --vcpus=1 \
    --os-type=linux \
    --os-variant=rhel7 \
    --initrd-inject=rhel.ks \
    --extra-args="ks=file:/rhel.ks console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH" \
    --disk=rhel.img,size=6,format=raw \
    --serial=pty \
    --nographics \
    --noreboot \
    --location=http://download.devel.redhat.com/released/RHEL-7/7.3/Server/ppc64le/os


(3) Boot the disk image using virtio-scsi:

qemu-system-ppc64 \
    -machine pseries \
    -cpu POWER8 -m 2048 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=rhel.img,snapshot=on,format=raw,if=none,id=hd0 \
    -device scsi-hd,drive=hd0 -serial stdio


(4) You should now observe the error during the first boot when the guest tries to
find its root disks.


To clean up after the above steps, you will need to do:

virsh destroy tmp
virsh undefine tmp
rm rhel.img

----

(In reply to Harald Hoyer from comment #16)
> hmm, /usr/lib/modules.d/90kernel-modules/module-setup.sh has
> 
>         # install virtual machine support
>         instmods virtio virtio_blk virtio_ring virtio_pci virtio_scsi \
>             "=drivers/pcmcia" =ide "=drivers/usb/storage"
> 
> 
> So, if the drivers are loaded on kickstart/installation time, they should be
> included in the final initramfs.

(File name is in fact /usr/lib/dracut/modules.d/90kernel-modules/module-setup.sh)

Yes, we have that file in the guest I built above, and it contains that
fragment too.  It seems to be conditional on `[[ -z $drivers ]]' and
nothing else.

It can't be running correctly because I pulled the initramfs from the guest above
and it does not contain virtio_scsi.ko

However the initramfs does contain virtio_blk.ko, virtio.ko, virtio_pci.ko and virtio_ring.ko.

(In reply to Harald Hoyer from comment #17)
> If you don't want the "hostonly" feature:
> 
> 
> %packages
> @core
> dracut-config-generic
> %end
> 
> which then should generate an initramfs with _all_ the kernel modules (and
> not just those, which were loaded on installation time)

Good tip, virt-builder should switch to doing this.

(In reply to Richard W.M. Jones from comment #18)
> I think what's needed first is a very clear and simple reproducer of
> the original bug.  This is the most minimal reproducer I could create:
> 
> (1) Create a file 'rhel.ks' containing:
> 
> install
> text
> reboot
> lang en_US.UTF-8
> keyboard us
> network --bootproto dhcp
> rootpw builder
> firewall --enabled --ssh
> timezone --utc America/New_York
> selinux --enforcing
> 
> bootloader --location=mbr --append="console=tty0 console=ttyS0,115200
> rd_NO_PLYMOUTH"
> 
> zerombr
> clearpart --all --initlabel
> autopart --type=lvm
> 
> # Halt the system once configuration has finished.
> poweroff
> 
> %packages
> @core
> %end
> 
> 
> (2) Run the following virt-install command (as non-root if you wish):
> 
> virt-install --name=tmp \
>     --ram=4096 \
>     --arch=ppc64le \
>     --machine=pseries \
>     --cpu=POWER8 \
>     --vcpus=1 \
>     --os-type=linux \
>     --os-variant=rhel7 \
>     --initrd-inject=rhel.ks \
>     --extra-args="ks=file:/rhel.ks console=tty0 console=ttyS0,115200
> rd_NO_PLYMOUTH" \
>     --disk=rhel.img,size=6,format=raw \
>     --serial=pty \
>     --nographics \
>     --noreboot \
>    
> --location=http://download.devel.redhat.com/released/RHEL-7/7.3/Server/
> ppc64le/os
> 
> 
> (3) Boot the disk image using virtio-scsi:
> 
> qemu-system-ppc64 \
>     -machine pseries \
>     -cpu POWER8 -m 2048 \
>     -device virtio-scsi-pci,id=scsi \
>     -drive file=rhel.img,snapshot=on,format=raw,if=none,id=hd0 \
>     -device scsi-hd,drive=hd0 -serial stdio
> 
> 
> (4) You should now observe the error during the first boot when the guest
> tries to
> find its root disks.

It works if you replace

  --disk=rhel.img,size=6,format=raw \

with

  --disk=rhel.img,size=6,format=raw,bus=scsi \
  --controller scsi,model=virtio-scsi \

in your virt-install command line: by default, the guest
will have a virtio-blk disk and dracut will only pick up
that one module, skipping virtio-scsi.

I also hit the issue on x86_64 server.
Here is beaker job: https://beaker.engineering.redhat.com/jobs/2690432
Here is console log: http://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2018/08/26904/2690432/5519832/console.log

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.