State of affairs:

qemu has a switch (-cpu ${name},host-phys-bits={on,off}).

When enabled (downstream default, also forced for microvm machine type upstream) standard cpuid query returns a physical address space size which is guaranteed to work (matches host by default, can be smaller, can not me larger).

When disabled (upstream default for q35 and pc) standard cpuid query returns whatever is configured (40 by default), even if the host supports less than that.

The problem the firmware has is that it can't trust the standard cpuid to return a usable value so the physical address space is not known.  Today the firmware tries to handle that by being conservative and not exceed the minimum size (phys-bits=36 aka 64G), which becomes increasingly problematic.

Two possible solutions I see for that:

(a) create a _reliable_ way for the guest to query the physical address space which is available.
(b) communicate the state of the host-phys-bits switch to the guest somehow, so the guest knows whenever standard cpuid query can be trusted or not.

> Today the
> firmware tries to handle that by being conservative and not exceed the
> minimum size (phys-bits=36 aka 64G), which becomes increasingly problematic.

For example when pci-assigning a gpu with a 32G memory bar (Cc'ing Guo Zhiyi).
Today the only option we have is to manually configure the firmware and override
the conservative defaults (-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536).

Would this be fixed if the non-hostphysbits case some how told you min(configured, actual) ?

(In reply to Dr. David Alan Gilbert from comment #6)
> Would this be fixed if the non-hostphysbits case some how told you
> min(configured, actual) ?

Yes.

We have that for the host-phys-bits=on case too, via host-phys-bits-limit.  You can configure the phys-bits presented to the guest.  It's required to be lower or equal the host value.  Mostly interesting for life migration compatibility, so one can configure the lowest value of the host pool.

So providing min(configured, actual) to the firmeare for the host-phys-bits=off case would be quite simliar, except for qemu not throwing an error in case the configured (or default) phys-bits are larger than actual.

Thanks, Julia.
In that case, we have to find someone who could take this up and move it forward otherwise we will have to keep it in the backlog.
Adding Igor to keep him in the loop.

Marking this as Triaged and keeping it in the backlog for now.
If this is something that we should be prioritizing please help by commenting or increasing the priority.

In the meantime, if anyone is interested please feel free to pick this up.

https://patchwork.ozlabs.org/project/qemu-devel/list/?series=316168

jira issue for firmware changes
https://issues.redhat.com/browse/RHELX-60

qemu v2: https://patchwork.ozlabs.org/project/qemu-devel/patch/20220908113109.470792-3-kraxel@redhat.com/

kernel: https://lore.kernel.org/kvm/20220908114146.473630-1-kraxel@redhat.com/

Hi Gerd, a couple of questions here.
- Would you like to own this BZ? Even if the patches comes via rebase after getting picked upstream we will need an assignee for any QE follow ups.
- Do we have a BZ tracking the kernel change?
Thanks

(In reply to Nitesh Narayan Lal from comment #15)
> Hi Gerd, a couple of questions here.
> - Would you like to own this BZ? Even if the patches comes via rebase after
> getting picked upstream we will need an assignee for any QE follow ups.

Yes, can take it.

> - Do we have a BZ tracking the kernel change?

Not needed, no kernel code changes.  It's really just reserving the bit so
it isn't grabbed for something else and upstream kernel repo is the place
where we have to do that because the master copy of the kvm header files
lives there.  Downstream kernel doesn't matter.

Ok, seems we have a plan now which doesn't require host (kernel/qemu) changes.

Quoting patch discussions:

> > Intel processors that are not extremely old have host-phys-bits equal
> > to 39, 46 or 52. Older processors that had 36, in all likelihood,
> > didn't have IOMMUs (so no big 64-bit BARs).
> >
> > AMD processors have had 48 for a while, though older consumer processors
> > had 40.
>
> How reliable is the vendorid?

Pretty reliable. In principle it can be changed, but there's no good reason
to do it (especially in a long lived VM) and it requires manual command
line intervention.

> Given newer processors have more than 40 and for older ones we know
> the possible values for the two relevant x86 vendors we could do
> something along the lines of:
>
>    phys-bits >= 41                   -> valid
>    phys-bits == 40    + AuthenticAMD -> valid
>    phys-bits == 36,39 + GenuineIntel -> valid
>    everything else                   -> invalid
>
> Does that look sensible to you?

Yes, it does!

No qemu changes needed-
edk2 progress is tracked here: https://issues.redhat.com/browse/RHEL-60
current state: merged upstream, stream9 / rhel-9.2 will get it with the next edk2 rebase.