Bug 1653140

Created attachment 1508481 [details]
Bug:failure at drivers...buf-messages-log

Created attachment 1508482 [details]
error:...vmlinuz-messages-log

Created attachment 1508483 [details]
client-guest-OVMF-log

Hi all, this are the steps for server-guest configuration. 

Boot two guests at same host. One as server and one as client. Both guests connect the provide switch. Configure the server first and then boot client. The client guest can be guided IPXE by server guest.

IPv4:

1. create a private switch on host

1.1)
#vim /etc/sysconfig/network-scripts/ifcfg-private-switch
DEVICE="private-switch"
NAME="private-switch"
ONBOOT="yes"
TYPE=bridge
BOOTPROTO=none
DEFROUTE=yes
IPADDR=192.168.1.101
PREFIX=24

1.2)
#vim /etc/qemu-ifup-private 

#!/bin/sh
switch=private-switch
/sbin/ifconfig $1 0.0.0.0 up
/usr/sbin/brctl addif ${switch} $1
/usr/sbin/brctl setfd ${switch} 0
/usr/sbin/brctl stp ${switch} off

# vim /etc/qemu-ifdown-private 

#!/bin/sh
switch=private-switch
/sbin/ifconfig $1 0.0.0.0 down
/usr/sbin/brctl delif ${switch} $1

1.3)
#systemctl restart network

[See the IP:192.168.1.101 is on the private bridge]


     
2. boot server guest on the private switch
     
    ipxe server:

/usr/libexec/qemu-kvm -M pc -cpu SandyBridge -nodefaults -smp 4 -m 4G -name IPXE-rhel8  -vga qxl -monitor stdio -boot menu=on -vnc :1 -drive file=IPXE-server-IPv4-goodbackup.qcow2,if=none,id=guest-img,format=qcow2,werror=stop,rerror=stop -device virtio-blk-pci,drive=guest-img,id=os-disk,bootindex=1 -device virtio-net-pci,netdev=tap11,mac=22:57:f8:dd:fe:3b,disable-legacy=on,disable-modern=off,bootindex=2 -netdev tap,id=tap11,script=/etc/qemu-ifup-private,downscript=/etc/qemu-ifdown-private -device virtio-net-pci,netdev=tap0,mac=02:02:a7:3f:64:30 -netdev tap,id=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

3. Configure IP address inside the guest:

#ifconfig ens4 192.168.1.100/24


[There are two NIC inside the guest, one is 10.66.XX.XX; the other one is 192.168.1.100/24] 



4.Configure tftp server inside the guest:
4.1) download tftp from brew:
#yum install tftp-server
#yum install tftp

#yum install xinetd

4.2) Create the tftp configure file:
# vim /etc/xinetd.d/tftp

# default: off
# description: The tftp server serves files using the trivial file transfer \
#       protocol.  The tftp protocol is often used to boot diskless \
#       workstations, download configuration files to network-aware printers, \
#       and to start the installation process for some operating systems.
service tftp
{
       disable                 = no
       socket_type             = dgram
       protocol                = udp
       wait                    = yes
       user                    = root
       server                  = /usr/sbin/in.tftpd
       server_args             = -v -v -v -v -v -s /var/lib/tftpboot
       per_source              = 11
       cps                     = 100 2
}

5. Config the shim
5.1) download shim from brew:
# yum -y install shim-x64-15-4.x86_64.rpm

cp /boot/efi/EFI/redhat/mmx64.efi  ==> /var/lib/tftpboot/
cp /boot/efi/EFI/redhat/shimx64-redhat.efi ==> /var/lib/tftpboot/
cp /boot/efi/EFI/redhat/shimx64.efi ==> /var/lib/tftpboot/

5.2) create images directory in /var/lib/tftpboot

5.3) download the vmlinuz and initrd.img to /var/lib/tftpboot/images/
#chmod 705 vmlinuz


6. config the grub
6.1) download grub2-efi-XXX from brew:
DO NOT install the "grub2-efi". This will change the structure of the grub2 directory structure. The guest can not be booted again. 

6.2) get the file "grubx64.efi"
#rpm2cpio grub2-efi-x64-2.02-58.el8.x86_64.rpm | pax -r -v

#cp "grubx64.efi" file to /var/lib/tftpboot/

#chmod 777 grubx64.efi

6.3) write the images
#vim grub.cfg

set timeout=60
    menuentry 'Install Red Hat Enterprise Linux 7.5'  {
        linuxefi images/vmlinuz ip=dhcp inst.repo=nfs:192.168.1.100:/home/RHEL-7.5-20180322.0-Server-x86_64-dvd1.iso
        initrdefi images/initrd.img
    }


6.4) Overall
-rw-r--r--  1 root root     230 Nov  8 16:52 grub.cfg
-rwxrwxrwx  1 root root 1735096 Nov  8 13:33 grubx64.efi
drwxr-xr-x. 2 root root      39 Nov  8 16:48 images
-rw-r--r--. 1 root root     163 Oct 29 15:00 ipxe.cfg
-rw-r--r--. 1 root root  720640 Oct 28 11:31 ipxe-x86_64.efi
-rwxr-xr-x. 1 root root 1160504 Oct 28 11:25 mmx64.efi
-rwxr-xr-x. 1 root root 1202952 Oct 28 11:26 shimx64.efi
-rwxr-xr-x. 1 root root 1205440 Oct 28 11:26 shimx64-redhat.efi
-rw-r--r--. 1 root root   71317 Oct 28 11:31 undionly.kpxe


Importantly, once all files were in place, I ran:

#chmod -cR u=rwX,g=rX,o=rX /var/lib/tftpboot/
#restorecon -FvvR /var/lib/tftpboot/


7. In order to share ISO file, setup nfs server inside guest
#yum -y install nfs-utils    
#vim /etc/exports
     
> /home *(rw,no_root_squash,async)

download the ISO in /home


8)Config DHCP:
#yum -y install dhcp-server
# vim /etc/dhcp/dhcpd.conf

# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp-server/dhcpd.conf.example
#   see dhcpd.conf(5) man page
#

 option architecture-type code 93 = unsigned integer 16;



    subnet 192.168.1.0 netmask 255.255.255.0 {
      option routers 192.168.1.100;
      option domain-name-servers 192.168.1.100;
      range 192.168.1.1 192.168.1.4;
      class "pxeclients" {
        match if substring (option vendor-class-identifier, 0, 9) =
          "PXEClient";
        next-server 192.168.1.100;
        if exists user-class and option user-class = "iPXE" {
          # rhbz-1462351
          # second stage: iPXE is booting, serve command script
          # filename "ipxe.cfg";
        } else {
          # first stage, firmware is booting
          if option architecture-type = 00:07 {
            # rhbz-1462351, serve iPXE
            filename "shimx64.efi";
          } elsif option architecture-type = 00:0b  {
            # serve arm64pe installer
            # filename "efi/boot/bootaa64.efi";
          } else {
            # rhbz-1462351, serve iPXE
            # filename "undionly.kpxe";
          }
        }
      }
    }

9) restart the server:
#systemctl stop firewalld
#setenforce 0
#iptables -F


#systemctl restart dhcpd
#systemctl restart xinetd
#systemctl restart nfs


Note:Regarding the "tftp.socket" and "tftp.service" systemd units, **DO NOT** enable or start those (contrarily to the Installation Guide instructions)


10) Created a new guest and let it it net-booted (ultimately) to the Anaconda welcome GUI.



ipxe client:
/usr/libexec/qemu-kvm -enable-kvm -M q35 -cpu SandyBridge -nodefaults -smp 4,cores=2,threads=2,sockets=1 -m 4G -name rhel7.5 -global driver=cfi.pflash01,property=secure,value=on -drive file=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/tmp/rhel7.5-IPXE-client-02/OVMF_VARS.fd,if=pflash,format=raw,unit=1,readonly=on -debugcon file:/home/test/client-02.log -global isa-debugcon.iobase=0x402 -drive file=/usr/share/OVMF/UefiShell.iso,if=none,cache=none,snapshot=off,aio=native,media=cdrom,id=cdrom1 -device ahci,id=ahci0 -device ide-cd,drive=cdrom1,id=ide-cd1,bus=ahci0.1 -vnc :3 -vga qxl -monitor stdio -drive file=IPXE-client-02.qcow2,if=none,id=guest-img,format=qcow2,werror=stop,rerror=stop -device virtio-blk-pci,drive=guest-img,id=os-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup-private,downscript=/etc/qemu-ifdown-private -device virtio-net-pci,netdev=hostnet0,id=net0,mac=24:be:05:15:d1:62 -serial unix:/tmp/console,server,nowait -boot menu=on,splash-time=50000


Consolution:
In this configuration,

      - OVMF successfully downloads and runs "shimx64.efi",
      - "shimx64.efi" downloads and interprets the command script "grubx64.efi",
      - "shimx64.efi" downloads the kernel and the initrd over TFTP, and
        launches the kernel.
      - The kernel finds the initial ramdisk because we explicitly tell it
        under what name to look for the ramdisk, with the "initrd=..."
        option. Grub sets this cmdline option automatically, with the
        "linuxefi" and "initrdefi" commands, but when using iPXE on an EFI
        system, the option has to be passed manually. Hat tip to
        <https://doc.rogerwhittaker.org.uk/ipxe-installation-and-EFI/> for
        the reminder.

Reference:
https://bugzilla.redhat.com/show_bug.cgi?id=1643377#c20

Hi,

According the comments#45 in Bug 1643377, the vhost-net should be disabled in the client-guest; However, When I set "vhost=off", the result is same. 


Hope this information will be helpful.

Michael, consider the following two changes to your setup:

- Change#1: for the "rpm2cpio" command, use the RHEL8 grub2-efi build that
  fixes BZ#1649048; that is, 2.02-61.el8.

- Change#2: append ",romfile=''" to  -device virtio-net-pci.

Can you reproduce the issue in the following *three* scenarios:

1. Only change#1 applied.
2. Only change#2 applied.
3. Both change#1 and change#2 applied.

Thanks!
Laszlo

Hi Laszlo:

Thank you for your reply. 

[1] I ONLY change packages RHEL8 grub2-efi to 2.02-61.el8. Get the new "grubx64.efi" file. The Bug can be reproduced. Thus, my understanding is that the new packages still does not work. Or that package does not seem to cause this issue. 

[2] The new packages and previous packages both can not work. Thus, I think it is not necessary to return the previous packages again. So I skip the scenarios#2. 

[3] I use the new grub2-efi packages and add the "romfile=''" in the client-guest. The client-guest can downloaded the file "shimx64.efi" successfully. Then it stop at the Fetching Netboot Image. There is the screenshot in the attachment. Also the full OVMF log in the attachment too.  

[4] If I add the "romfile=''" in the sever-guest. There is nothing different with original bug. 


Hope this information can give some help. 

Thanks

Created attachment 1509346 [details]
screenshot PXE when add "romfile=' ' "

Created attachment 1509347 [details]
OVMF log when add "romfile=' ' "

Hi Michael, your results confirm that OVMF's built-in VirtioNetDxe driver was used instead of the virtio-net driver from the ipxe-roms-qemu package. This is the effect of passing the "romfile=''" option to the client guest.

However, based on the ASSERT() at the end of the log in comment 9, I'm having doubts about you using a grub2-efi binary with bug 1649048 fixed. The ASSERT() is the precise symptom of bug 1649048.

Is it possible that shim actually downloaded grub successfully, and grub triggered the error? I can't tell that from the logs that you attached, but if you look at the TFTP server log (on the server guest), you'll see whether there were any TFTP requests made by grub2. Thanks.

(BTW I tried to verify bug 1649048 yesterday myself, but now for some reason I seem unable to reproduce it in the first place.)

I may have to log in to your repro machine. Thanks.

Hi Michael,

actually, I can reproduce the issue from comment 9 / bug 1649048, and I agree: grub2-efi 2.02-61.el8 does not fix bug 1649048. In other words, I confirm your point [3] from comment 7.

And, on my end, I can also look at the TFTP server log, and I see the issue still hits from within grub2:

Nov 28 14:21:08 ovmf-rhel7-q35 in.tftpd[2574]: RRQ from ::ffff:192.168.124.104 filename shimx64.efi
Nov 28 14:21:08 ovmf-rhel7-q35 in.tftpd[2574]: Client ::ffff:192.168.124.104 finished shimx64.efi
Nov 28 14:21:08 ovmf-rhel7-q35 in.tftpd[2575]: RRQ from ::ffff:192.168.124.104 filename grubx64.efi
Nov 28 14:21:09 ovmf-rhel7-q35 in.tftpd[2575]: Client ::ffff:192.168.124.104 finished grubx64.efi

Michael,

I believe to have found another issue in grub2 (RHEL8). I'm going to open a new bug about that. I think it could explain the behavior we are seeing with both ipxe-roms-qemu, and the built-in driver VirtioNetDxe. So, for now, I'm going to mark this BZ as TestOnly, and make it dependent on the new grub2 bug.

Thanks
Laszlo

(In reply to Laszlo Ersek from comment #12)
> Michael,
> 
> I believe to have found another issue in grub2 (RHEL8). I'm going to open a
> new bug about that. I think it could explain the behavior we are seeing with
> both ipxe-roms-qemu, and the built-in driver VirtioNetDxe. So, for now, I'm
> going to mark this BZ as TestOnly, and make it dependent on the new grub2
> bug.
> 
> Thanks
> Laszlo


Hi Laszlo: 
Thanks for your help.

Hi all:

Depends on Bug 1654388, the new fixed In Version: grub2-2.02-62.el8. 

Thus, using package "grub2-efi-x64-2.02-62.el8.x86_64.rpm" instead of the "grub2-efi-x64-2.02-61.el8.x86_64.rpm". Then repeat the test follow the step in comment#4. 

The issue is gone. I can see the installation welcome UI window in the client guest. PXE-IPv4-fireware+shim+grub2 net boot working well. Thus, marking this BZ as Verified. 


Thanks

*** This bug has been marked as a duplicate of bug 1654388 ***