Bug 1789797

I'd like to clarify the scope of this BZ (and the goals of the upstream
patch series, which I posted at:

[edk2-devel] [PATCH 0/2] UefiBootManagerLib, HttpDxe: tweaks for large HTTP(S) downloads
https://edk2.groups.io/g/devel/message/53034
http://mid.mail-archive.com/20200108234313.28510-1-lersek@redhat.com
)

There are two failure scenarios. Comment#0 currently mixes them up, so
let me isolate one scenario from the other.


(1) The first symptom is when the net-client VM's RAM is simply too
small for accepting the remote file (such as a very large ISO image)
over HTTPS Boot. In this case, the HTTPS Boot attempt in the net-client
VM is aborted very quickly, and there is basically nothing usable
printed to *either* the UEFI console (graphical screen and/or serial
port), *or* the OVMF debug log. The user is simply returned to the UEFI
Setup TextUI, almost immediately. Therefore this situation is difficult
to identify / diagnose.

One hint about this particular problem is the net-server VM's apache log
("/var/log/httpd/ssl_request_log"). When this failure occurs, there will
only be a HEAD request in the log, and no GET request. That's because
OVMF, running on the net-client VM, sends a HEAD request for learning
the size of the file to download. Then the memory allocation fails, and
so there is no GET request. If there is enough RAM in the net-client VM,
then the HEAD request is followed by a GET request, in the net-server
VM's "/var/log/httpd/ssl_request_log" file.

This issue is remedied by the first patch in the series. Obviously the
HTTPS Boot can still not succeed (there still isn't enough RAM in
net-client), but the OVMF debug log will now contain a single-line
message like:

> UiApp:BmExpandLoadFile: failed to allocate reserved pages:
> BufferSize=4501536768
> LoadFile="PciRoot(0x0)/Pci(0x3,0x0)/MAC(5254001B103E,0x1)/IPv4(0.0.0.0,TCP,DHCP,192.168.124.106,192.168.124.1,255.255.255.0)/Dns(192.168.124.1)/Uri(https://ipv4-server/RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso)"
> FilePath=""

Here, "BufferSize" stands for the remote file size, and the Uri() device
path node contains the URL of the file to download (which was provided
to net-client by net-server's DHCP server)


(2) The other failure scenario is different. In this case, there are two
conditions:

(2a) the size of the remote file equals 4 GiB *plus* an integral
multiple of 16 KiB;

(2b) the RAM in the net-client VM *is* sufficient for downloading that
large file (for example 8GiB or 12GiB could be the net-client VM's RAM
size).

To given an example for (2a), consider
"RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso". Its size is 4 GiB plus 197
MiB. Because 197 MiB is a whole multiple of 16 KiB, it satisfies the
requirement (for reproducing the issue).

The symptom of this HTTPS Boot failure is that only an initial slice of
the remote file is downloaded, and when exactly 4GiB are *left* to
download -- that is, in the above example, after 197 MiB has been
downloaded --, the download is aborted, with "Error: Unexpected network
error".

This issue is solved by the second patch in the series. The expected
result is that the HTTPS Boot simply succeeds.


So, in order for virt-QE to verify this BZ, two cases should be
reproduced / checked:

- Remote file too large for net-client's RAM: check for the informative
message in the OVMF log.

- Remote file *not* too large for net-client's RAM, but has a size
(4GiB+N*16KiB): check that the download simply works.

Thanks.

(In reply to Laszlo Ersek from comment #1)

> [edk2-devel] [PATCH 0/2] UefiBootManagerLib, HttpDxe: tweaks for large HTTP(S) downloads
> https://edk2.groups.io/g/devel/message/53034
> http://mid.mail-archive.com/20200108234313.28510-1-lersek@redhat.com

Merged upstream as commit range b112ec225f1c..4cca7923992a.

QA_ACK, please?

Laszlo,

Thanks for your detailed explanation. 


I reproduced them on edk2-ovmf-20190829git37eef91017ad-4.el8.noarch.
Retested them on edk2-ovmf-20190829git37eef91017ad-5.el8.noarch, all work well, so set bug status to VERIFIED.


Details:

Versions:
kernel-4.18.0-167.el8.x86_64
qemu-kvm-4.2.0-6.module+el8.2.0+5451+991cea0d
edk2-ovmf-20190829git37eef91017ad-5.el8.noarch
openssl-1.1.1c-10.el8.x86_64


Download RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso and place it under "/var/www/html" in net-server virtual machine. 
Try to install it with UEFI IPv4 and UEFI IPv6.
For details steps, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1536624#c60.



case 1.  set memory to 2Gib

e.g.
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory> 

(1) reproduced it on edk2-ovmf-20190829git37eef91017ad-4.el8.noarch
 
check HEAD request and GET request in "/var/log/httpd/ssl_request_log" on the net-server virtual machine.

In this case, there was only a HEAD request, and no GET request. There is no explicit error message about this in the OVMF
log. 


(2) tested on edk2-ovmf-20190829git37eef91017ad-5.el8.noarch, it has been fixed.

  Found the informative message in the OVMF log:
  For IPv4:
  UiApp:BmExpandLoadFile: failed to allocate reserved pages: BufferSize=4501536768 LoadFile="PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(525400DD90E9,0x1)/IPv4(0.0.0.0,TCP,DHCP,192.168.124.101,192.168.124.1,255.255.255.0)/Dns(192.168.124.1)/Uri(https://ipv4-server/RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso)" FilePath=""

  For IPv6:
  UiApp:BmExpandLoadFile: failed to allocate reserved pages: BufferSize=4501536768 LoadFile="PciRoot(0x0)/Pci(0x2,0x0)/Pci(0x0,0x0)/MAC(525400DD90E9,0x1)/IPv6(FD33:EB1B:9B36:0000:0000:0000:0000:0002,TCP,Static,FD33:EB1B:9B36:0000:0000:0000:0000:00C8,0x40,FE80:0000:0000:0000:5054:00FF:FE28:8AFF)/Dns(FD33:EB1B:9B36:0000:0000:0000:0000:0001)/Uri(https://ipv6-server/RHEL-7.7-20190723.1-Server-x86_64-dvd1.iso)" FilePath=""



case 2. set memory to 12Gib 

e.g.
<memory unit='KiB'>12582912</memory>
<currentMemory unit='KiB'>12582912</currentMemory>

(1)  reproduced it on edk2-ovmf-20190829git37eef91017ad-4.el8.noarch

  HTTPS Boot failure is that only an initial slice of the remote file is downloaded,
  and when exactly 4GiB are left to download, after 197 MiB has been downloaded, 
  the download is aborted, with "Error: Unexpected network error".
  
  Please refer to attachment for screenshot.


(2) tested on edk2-ovmf-20190829git37eef91017ad-5.el8.noarch, it has been fixed.
  The remote file is downloaded successfully and it is installed successfully with UEFI IPv4 and UEFI IPv6.
  
  Please refer to attachment for screenshot.

Created attachment 1653827 [details]
unexpected_network_error

Created attachment 1653828 [details]
ipv4_downloading

Created attachment 1653829 [details]
ipv4_installation

Created attachment 1653830 [details]
ipv6_downloading

Created attachment 1653831 [details]
ipv6_installation

Xueqiang Wei, the results look good to me, many thanks.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1712