Bug 1653140 - failure at drivers/bus/virtio-ring.c:69/vring_get_buf() when using pxe boot on OVMF guest.
Summary: failure at drivers/bus/virtio-ring.c:69/vring_get_buf() when using pxe boot o...
Keywords:
Status: CLOSED DUPLICATE of bug 1654388
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: grub2
Version: 8.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: Laszlo Ersek
QA Contact: FuXiangChun
URL:
Whiteboard:
Depends On: 1654388
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-26 03:30 UTC by Michael
Modified: 2018-12-05 13:57 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-05 13:57:46 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
Bug:failure at drivers...buf-messages-log (2.81 KB, text/plain)
2018-11-26 04:18 UTC, Michael
no flags Details
error:...vmlinuz-messages-log (10.13 KB, text/plain)
2018-11-26 04:19 UTC, Michael
no flags Details
client-guest-OVMF-log (478.68 KB, text/plain)
2018-11-26 04:21 UTC, Michael
no flags Details
screenshot PXE when add "romfile=' ' " (17.67 KB, image/png)
2018-11-28 02:58 UTC, Michael
no flags Details
OVMF log when add "romfile=' ' " (477.58 KB, text/plain)
2018-11-28 02:59 UTC, Michael
no flags Details

Description Michael 2018-11-26 03:30:13 UTC
Description of problem:

According to Bug 1643377, try to use firmware+shim+grub2 booting the OVMF guest. I can see the "Install Red Hat Enterprise Linux **" window in the OVMF guest. It also shows the "Press 'e' to edit the selected item, or 'c' for a command prompt" 

However, the issue is happened when enter the installation process. Some times the guest hang; Some times appear bug information as follow:

> BUG: failure at drivers/bus/virtio-ring.c:69/vring_get_buf()!

Some times appear file "vmlinuz" or "initrd.img" error information as follow:

> error: ../../grub-core/net/tftp.c:450:time out opening `images/vmlinuz'.
> error: ../../grub-core/loader/i386/efi/linux.c:93:you need to load the kernel
>first.


Version-Release number of selected component (if applicable):
kernel: 4.18.0-40.el8.x86_64
qemu-kvm-3.0.0-2.module+el8+2208+e41b12e0.x86_64
ipxe-roms-qemu-20170710-5.git0600d3ae.module+el8+2179+85112f94.noarch
edk2-ovmf-20180508gitee3198e672e2-8.el8.noarch


How reproducible:
100%


Steps to Reproduce:
1. Boot the server-guest; add IP address inside the guest; restart dhcpd; xinetd; nfs server. 

2. Boot the client-guest; select "UEFI-PXEv4 booting". 
3. Maybe need to wait one or two minutes. 


Actual results:
[1] Guest Hang;
[2] BUG: failure at drivers/bus/virtio-ring.c:69/vring_get_buf()!
[3] error: ../../grub-core/net/tftp.c:450:time out opening `images/vmlinuz'.
    error: ../../grub-core/loader/i386/efi/linux.c:93:you need to load the     kernel first.


Expected results:
Guest boot successfully and I can see the UI installation Welcome window. 


Additional info:

[1] The issue I am encountering is specific to the interaction between RHEL8
grub2 and RHEL7 ipxe-roms-qemu. The detail analysis please see comments:
https://bugzilla.redhat.com/show_bug.cgi?id=1643377#c45

[2] This is net-booting OVMF guest case via PXE. But Raycom office does not have  PXE dhcp server. Thus, two guests at same host. One as server and one as client.  The client guest can be guided PXE by server guest. The follow comments I will shows the steps of server-guest configuration. 

[3] I have a server-guest which already finish the all configuration. any one can use it straight way. 
http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/personal/choma/PXE-IPv4-firmware+shim+grub2-choma.qcow2

[4] Some log is in the attachment. Different error information I upload different log.

Comment 1 Michael 2018-11-26 04:18:43 UTC
Created attachment 1508481 [details]
Bug:failure at drivers...buf-messages-log

Comment 2 Michael 2018-11-26 04:19:30 UTC
Created attachment 1508482 [details]
error:...vmlinuz-messages-log

Comment 3 Michael 2018-11-26 04:21:04 UTC
Created attachment 1508483 [details]
client-guest-OVMF-log

Comment 4 Michael 2018-11-26 04:30:46 UTC
Hi all, this are the steps for server-guest configuration. 

Boot two guests at same host. One as server and one as client. Both guests connect the provide switch. Configure the server first and then boot client. The client guest can be guided IPXE by server guest.

IPv4:

1. create a private switch on host

1.1)
#vim /etc/sysconfig/network-scripts/ifcfg-private-switch
DEVICE="private-switch"
NAME="private-switch"
ONBOOT="yes"
TYPE=bridge
BOOTPROTO=none
DEFROUTE=yes
IPADDR=192.168.1.101
PREFIX=24

1.2)
#vim /etc/qemu-ifup-private 

#!/bin/sh
switch=private-switch
/sbin/ifconfig $1 0.0.0.0 up
/usr/sbin/brctl addif ${switch} $1
/usr/sbin/brctl setfd ${switch} 0
/usr/sbin/brctl stp ${switch} off

# vim /etc/qemu-ifdown-private 

#!/bin/sh
switch=private-switch
/sbin/ifconfig $1 0.0.0.0 down
/usr/sbin/brctl delif ${switch} $1

1.3)
#systemctl restart network

[See the IP:192.168.1.101 is on the private bridge]


     
2. boot server guest on the private switch
     
    ipxe server:

/usr/libexec/qemu-kvm -M pc -cpu SandyBridge -nodefaults -smp 4 -m 4G -name IPXE-rhel8  -vga qxl -monitor stdio -boot menu=on -vnc :1 -drive file=IPXE-server-IPv4-goodbackup.qcow2,if=none,id=guest-img,format=qcow2,werror=stop,rerror=stop -device virtio-blk-pci,drive=guest-img,id=os-disk,bootindex=1 -device virtio-net-pci,netdev=tap11,mac=22:57:f8:dd:fe:3b,disable-legacy=on,disable-modern=off,bootindex=2 -netdev tap,id=tap11,script=/etc/qemu-ifup-private,downscript=/etc/qemu-ifdown-private -device virtio-net-pci,netdev=tap0,mac=02:02:a7:3f:64:30 -netdev tap,id=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

3. Configure IP address inside the guest:

#ifconfig ens4 192.168.1.100/24


[There are two NIC inside the guest, one is 10.66.XX.XX; the other one is 192.168.1.100/24] 



4.Configure tftp server inside the guest:
4.1) download tftp from brew:
#yum install tftp-server
#yum install tftp

#yum install xinetd

4.2) Create the tftp configure file:
# vim /etc/xinetd.d/tftp

# default: off
# description: The tftp server serves files using the trivial file transfer \
#       protocol.  The tftp protocol is often used to boot diskless \
#       workstations, download configuration files to network-aware printers, \
#       and to start the installation process for some operating systems.
service tftp
{
       disable                 = no
       socket_type             = dgram
       protocol                = udp
       wait                    = yes
       user                    = root
       server                  = /usr/sbin/in.tftpd
       server_args             = -v -v -v -v -v -s /var/lib/tftpboot
       per_source              = 11
       cps                     = 100 2
}

5. Config the shim
5.1) download shim from brew:
# yum -y install shim-x64-15-4.x86_64.rpm

cp /boot/efi/EFI/redhat/mmx64.efi  ==> /var/lib/tftpboot/
cp /boot/efi/EFI/redhat/shimx64-redhat.efi ==> /var/lib/tftpboot/
cp /boot/efi/EFI/redhat/shimx64.efi ==> /var/lib/tftpboot/

5.2) create images directory in /var/lib/tftpboot

5.3) download the vmlinuz and initrd.img to /var/lib/tftpboot/images/
#chmod 705 vmlinuz


6. config the grub
6.1) download grub2-efi-XXX from brew:
DO NOT install the "grub2-efi". This will change the structure of the grub2 directory structure. The guest can not be booted again. 

6.2) get the file "grubx64.efi"
#rpm2cpio grub2-efi-x64-2.02-58.el8.x86_64.rpm | pax -r -v

#cp "grubx64.efi" file to /var/lib/tftpboot/

#chmod 777 grubx64.efi

6.3) write the images
#vim grub.cfg

set timeout=60
    menuentry 'Install Red Hat Enterprise Linux 7.5'  {
        linuxefi images/vmlinuz ip=dhcp inst.repo=nfs:192.168.1.100:/home/RHEL-7.5-20180322.0-Server-x86_64-dvd1.iso
        initrdefi images/initrd.img
    }


6.4) Overall
-rw-r--r--  1 root root     230 Nov  8 16:52 grub.cfg
-rwxrwxrwx  1 root root 1735096 Nov  8 13:33 grubx64.efi
drwxr-xr-x. 2 root root      39 Nov  8 16:48 images
-rw-r--r--. 1 root root     163 Oct 29 15:00 ipxe.cfg
-rw-r--r--. 1 root root  720640 Oct 28 11:31 ipxe-x86_64.efi
-rwxr-xr-x. 1 root root 1160504 Oct 28 11:25 mmx64.efi
-rwxr-xr-x. 1 root root 1202952 Oct 28 11:26 shimx64.efi
-rwxr-xr-x. 1 root root 1205440 Oct 28 11:26 shimx64-redhat.efi
-rw-r--r--. 1 root root   71317 Oct 28 11:31 undionly.kpxe


Importantly, once all files were in place, I ran:

#chmod -cR u=rwX,g=rX,o=rX /var/lib/tftpboot/
#restorecon -FvvR /var/lib/tftpboot/


7. In order to share ISO file, setup nfs server inside guest
#yum -y install nfs-utils    
#vim /etc/exports
     
> /home *(rw,no_root_squash,async)

download the ISO in /home


8)Config DHCP:
#yum -y install dhcp-server
# vim /etc/dhcp/dhcpd.conf

# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp-server/dhcpd.conf.example
#   see dhcpd.conf(5) man page
#

 option architecture-type code 93 = unsigned integer 16;



    subnet 192.168.1.0 netmask 255.255.255.0 {
      option routers 192.168.1.100;
      option domain-name-servers 192.168.1.100;
      range 192.168.1.1 192.168.1.4;
      class "pxeclients" {
        match if substring (option vendor-class-identifier, 0, 9) =
          "PXEClient";
        next-server 192.168.1.100;
        if exists user-class and option user-class = "iPXE" {
          # rhbz-1462351
          # second stage: iPXE is booting, serve command script
          # filename "ipxe.cfg";
        } else {
          # first stage, firmware is booting
          if option architecture-type = 00:07 {
            # rhbz-1462351, serve iPXE
            filename "shimx64.efi";
          } elsif option architecture-type = 00:0b  {
            # serve arm64pe installer
            # filename "efi/boot/bootaa64.efi";
          } else {
            # rhbz-1462351, serve iPXE
            # filename "undionly.kpxe";
          }
        }
      }
    }

9) restart the server:
#systemctl stop firewalld
#setenforce 0
#iptables -F


#systemctl restart dhcpd
#systemctl restart xinetd
#systemctl restart nfs


Note:Regarding the "tftp.socket" and "tftp.service" systemd units, **DO NOT** enable or start those (contrarily to the Installation Guide instructions)


10) Created a new guest and let it it net-booted (ultimately) to the Anaconda welcome GUI.



ipxe client:
/usr/libexec/qemu-kvm -enable-kvm -M q35 -cpu SandyBridge -nodefaults -smp 4,cores=2,threads=2,sockets=1 -m 4G -name rhel7.5 -global driver=cfi.pflash01,property=secure,value=on -drive file=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/tmp/rhel7.5-IPXE-client-02/OVMF_VARS.fd,if=pflash,format=raw,unit=1,readonly=on -debugcon file:/home/test/client-02.log -global isa-debugcon.iobase=0x402 -drive file=/usr/share/OVMF/UefiShell.iso,if=none,cache=none,snapshot=off,aio=native,media=cdrom,id=cdrom1 -device ahci,id=ahci0 -device ide-cd,drive=cdrom1,id=ide-cd1,bus=ahci0.1 -vnc :3 -vga qxl -monitor stdio -drive file=IPXE-client-02.qcow2,if=none,id=guest-img,format=qcow2,werror=stop,rerror=stop -device virtio-blk-pci,drive=guest-img,id=os-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup-private,downscript=/etc/qemu-ifdown-private -device virtio-net-pci,netdev=hostnet0,id=net0,mac=24:be:05:15:d1:62 -serial unix:/tmp/console,server,nowait -boot menu=on,splash-time=50000


Consolution:
In this configuration,

      - OVMF successfully downloads and runs "shimx64.efi",
      - "shimx64.efi" downloads and interprets the command script "grubx64.efi",
      - "shimx64.efi" downloads the kernel and the initrd over TFTP, and
        launches the kernel.
      - The kernel finds the initial ramdisk because we explicitly tell it
        under what name to look for the ramdisk, with the "initrd=..."
        option. Grub sets this cmdline option automatically, with the
        "linuxefi" and "initrdefi" commands, but when using iPXE on an EFI
        system, the option has to be passed manually. Hat tip to
        <https://doc.rogerwhittaker.org.uk/ipxe-installation-and-EFI/> for
        the reminder.

Reference:
https://bugzilla.redhat.com/show_bug.cgi?id=1643377#c20

Comment 5 Michael 2018-11-26 05:25:34 UTC
Hi,

According the comments#45 in Bug 1643377, the vhost-net should be disabled in the client-guest; However, When I set "vhost=off", the result is same. 


Hope this information will be helpful.

Comment 6 Laszlo Ersek 2018-11-27 21:02:19 UTC
Michael, consider the following two changes to your setup:

- Change#1: for the "rpm2cpio" command, use the RHEL8 grub2-efi build that
  fixes BZ#1649048; that is, 2.02-61.el8.

- Change#2: append ",romfile=''" to  -device virtio-net-pci.

Can you reproduce the issue in the following *three* scenarios:

1. Only change#1 applied.
2. Only change#2 applied.
3. Both change#1 and change#2 applied.

Thanks!
Laszlo

Comment 7 Michael 2018-11-28 02:56:49 UTC
Hi Laszlo:

Thank you for your reply. 

[1] I ONLY change packages RHEL8 grub2-efi to 2.02-61.el8. Get the new "grubx64.efi" file. The Bug can be reproduced. Thus, my understanding is that the new packages still does not work. Or that package does not seem to cause this issue. 

[2] The new packages and previous packages both can not work. Thus, I think it is not necessary to return the previous packages again. So I skip the scenarios#2. 

[3] I use the new grub2-efi packages and add the "romfile=''" in the client-guest. The client-guest can downloaded the file "shimx64.efi" successfully. Then it stop at the Fetching Netboot Image. There is the screenshot in the attachment. Also the full OVMF log in the attachment too.  

[4] If I add the "romfile=''" in the sever-guest. There is nothing different with original bug. 


Hope this information can give some help. 

Thanks

Comment 8 Michael 2018-11-28 02:58:15 UTC
Created attachment 1509346 [details]
screenshot PXE when add "romfile=' ' "

Comment 9 Michael 2018-11-28 02:59:21 UTC
Created attachment 1509347 [details]
OVMF log when add "romfile=' ' "

Comment 10 Laszlo Ersek 2018-11-28 12:52:37 UTC
Hi Michael, your results confirm that OVMF's built-in VirtioNetDxe driver was used instead of the virtio-net driver from the ipxe-roms-qemu package. This is the effect of passing the "romfile=''" option to the client guest.

However, based on the ASSERT() at the end of the log in comment 9, I'm having doubts about you using a grub2-efi binary with bug 1649048 fixed. The ASSERT() is the precise symptom of bug 1649048.

Is it possible that shim actually downloaded grub successfully, and grub triggered the error? I can't tell that from the logs that you attached, but if you look at the TFTP server log (on the server guest), you'll see whether there were any TFTP requests made by grub2. Thanks.

(BTW I tried to verify bug 1649048 yesterday myself, but now for some reason I seem unable to reproduce it in the first place.)

I may have to log in to your repro machine. Thanks.

Comment 11 Laszlo Ersek 2018-11-28 13:39:29 UTC
Hi Michael,

actually, I can reproduce the issue from comment 9 / bug 1649048, and I agree: grub2-efi 2.02-61.el8 does not fix bug 1649048. In other words, I confirm your point [3] from comment 7.

And, on my end, I can also look at the TFTP server log, and I see the issue still hits from within grub2:

Nov 28 14:21:08 ovmf-rhel7-q35 in.tftpd[2574]: RRQ from ::ffff:192.168.124.104 filename shimx64.efi
Nov 28 14:21:08 ovmf-rhel7-q35 in.tftpd[2574]: Client ::ffff:192.168.124.104 finished shimx64.efi
Nov 28 14:21:08 ovmf-rhel7-q35 in.tftpd[2575]: RRQ from ::ffff:192.168.124.104 filename grubx64.efi
Nov 28 14:21:09 ovmf-rhel7-q35 in.tftpd[2575]: Client ::ffff:192.168.124.104 finished grubx64.efi

Comment 12 Laszlo Ersek 2018-11-28 14:52:18 UTC
Michael,

I believe to have found another issue in grub2 (RHEL8). I'm going to open a new bug about that. I think it could explain the behavior we are seeing with both ipxe-roms-qemu, and the built-in driver VirtioNetDxe. So, for now, I'm going to mark this BZ as TestOnly, and make it dependent on the new grub2 bug.

Thanks
Laszlo

Comment 13 Michael 2018-11-29 05:10:25 UTC
(In reply to Laszlo Ersek from comment #12)
> Michael,
> 
> I believe to have found another issue in grub2 (RHEL8). I'm going to open a
> new bug about that. I think it could explain the behavior we are seeing with
> both ipxe-roms-qemu, and the built-in driver VirtioNetDxe. So, for now, I'm
> going to mark this BZ as TestOnly, and make it dependent on the new grub2
> bug.
> 
> Thanks
> Laszlo


Hi Laszlo: 
Thanks for your help.

Comment 18 Michael 2018-12-05 01:09:27 UTC
Hi all:

Depends on Bug 1654388, the new fixed In Version: grub2-2.02-62.el8. 

Thus, using package "grub2-efi-x64-2.02-62.el8.x86_64.rpm" instead of the "grub2-efi-x64-2.02-61.el8.x86_64.rpm". Then repeat the test follow the step in comment#4. 

The issue is gone. I can see the installation welcome UI window in the client guest. PXE-IPv4-fireware+shim+grub2 net boot working well. Thus, marking this BZ as Verified. 


Thanks

Comment 19 Laszlo Ersek 2018-12-05 13:57:46 UTC

*** This bug has been marked as a duplicate of bug 1654388 ***


Note You need to log in before you can comment on or make changes to this bug.