Libvirt already supports the "virtio-gpu-pci" device for AARCH64 guests (as the "virtio" model), but that's good enough only for Linux guests, not for Windows guests. Windows requires a directly accessible framebuffer, and virtio-gpu-pci doesn't provide one. QEMU implements (and libvirt supports) a number of VGA-like display devices, which do offer framebuffers. Such as standard VGA, virtio-vga (which is virtio-gpu-pci plus VGA), and QXL. The problem with all of these is the following (on AARCH64 only): When using VGA, the framebuffer used by the guest lives inside a (virtual) PCI MMIO BAR. Accordingly, the guest maps the framebuffer as uncacheable. On the host side though, QEMU (and the host kernel) treat the virtual framebuffer as normal memory. Therefore on the host side, the framebuffer is mapped as write-back cacheable. This means that the guest and the host have different mappings (different cacheability attribtues) for the same memory area. On x86_64 hosts, this is no problem, because the host side mapping takes effect (overriding the guest-side mapping), so ultimately both guest and host access the framebuffer (which lives in normal DRAM) as write-back cacheable. This means that both guest and host go through the PCPU's data cache(s) first, and they are coherent with each other. On aarch64 hosts, this is not the case. Because the guest uses UC mapping, when it writes to the framebuffer, it avoids the PCPU caches, and goes directly to DRAM. On the host side however, QEMU reads the framebuffer through a WB mapping, and so it goes through the PCPU caches. The view is not coherent, and you get display corruption (stale data displayed by QEMU from the PCPU cache). AIUI, the ARMv8.4 architecture will finally mitigate this, by introducing a new feature bit (I think it's called FWB?) that will make its behavior similar to x86-64's. Until then however, the "framebuffer in PCI MMIO BAR" approach of VGA remains unusable on aarch64/KVM. This is why we killed the QemuVideoDxe driver in the edk2/ArmVirtQemu firmware. On QEMU/TCG (i.e., software emulation), this issue is unobservable, as in that case, only one mapping exists (the host side one). The issue also goes away with virtio-gpu-pci, because (a) there is no framebuffer to speak of, (b) all display data, exchanged with virtio transfers, are mapped as normal DRAM (WB) by both guest and host. Unfortunately, Windows absolutely insist on a framebuffer for booting (even though a direct access framebuffer is optional, according to the UEFI spec). Therefore we can't use virtio-gpu-pci. (Again, don't confuse virtio-gpu-pci with virtio-vga. The latter is virtio-gpu-pci PLUS the traditional VGA stuff. No good for aarch64.) The issue also goes away with the "ramfb" device model that Gerd wrote for QEMU. It is a (comparatively) simple platform device that uses DRAM (WB) as the framebuffer, so again, both sides' mappings agree, wrt. cacheability. The UEFI driver (QemuRamfbDxe, also by Gerd) exposes the direct-access framebuffer to the Windows boot loader, so the framebuffer requirement is satisfied, while also side-stepping the non-coherence issue on ARM64. Ramfb is not a PCI device, but it doesn't need to be. UEFI-booted Windows is expected to work just with the UEFI framebuffer that it inherits. Graphics mode changes and such are not expected to work, of course, but basic display is expected to work. Please introduce the "ramfb" display device model in the domain XML. Thanks!
(In reply to Laszlo Ersek from comment #0) > When using VGA, the framebuffer used by the guest lives inside a > (virtual) PCI MMIO BAR. Accordingly, the guest maps the framebuffer as > uncacheable. I'm curious why is this problem specific to VGA memory? Is that because VGA is the only device mapping memory that way on QEMU? (I know this is rare, I do not know for certain it is unique) How does QEMU deal with other devices if it can't assume it sees valid data for any guest MMIO or UC mappings? If the architecture does not ensure coherency when one mapping is UC and the other is WB, doesn't that imply that QEMU should be able to override UC for some mapping that are known to be backed by real memory? Or alternatively, to invalidate the cache in some other way before accessing VGA memory. (My concern here is that the ramfb approach would not fix the VGA bug, and I find it likely that someone might still misconfigure QEMU)
(In reply to Christophe de Dinechin from comment #1) > (In reply to Laszlo Ersek from comment #0) > > When using VGA, the framebuffer used by the guest lives inside a > > (virtual) PCI MMIO BAR. Accordingly, the guest maps the framebuffer as > > uncacheable. > > I'm curious why is this problem specific to VGA memory? > Is that because VGA is the only device mapping memory that way on QEMU? > (I know this is rare, I do not know for certain it is unique) Normally a GPA range that appears as writeable MMIO to the guest is not backed by any host memory. There is no memory slot in KVM, and every write access traps separately. QEMU provides the device emulation on an individual write access basis. VGA is different; it is backed by writeable host memory (and a r/w KVM memslot), and QEMU uses something called "coalesced MMIO" to refresh the display from that host RAM area. My understanding is quite lacking in this area, but VGA framebuffer writes from the guest do not trap individually. > How does QEMU deal with other devices if it can't assume it sees valid data > for any guest MMIO or UC mappings? For other devices, the specifics of the MMIO write are passed down to QEMU from KVM based on the trap symptoms. QEMU doesn't have to read RAM for the details. > If the architecture does not ensure coherency when one mapping is UC and > the other is WB, doesn't that imply that QEMU should be able to override UC > for some mapping that are known to be backed by real memory? This was one of the ideas discussed (multiple times); KVM could perhaps expose an ioctl() just for this. Ultimately the idea was rejected. I don't remember why. > Or alternatively, to invalidate the cache in some other way before accessing VGA memory. This was another idea that ended up being rejected. It would be a privileged operation, for starters. (QEMU runs restricted so it couldn't directly execute the necessary instructions. There could be other obstacles. I don't remember.) > (My concern here is that the ramfb approach would not fix the VGA bug, > and I find it likely that someone might still misconfigure QEMU) ramfb is not meant to fix the VGA bug; ramfb is meant to replace VGA. The ramfb framebuffer is exposed as normal RAM (not MMIO) to the guest, hence the guest is expected to map it with WB attributes. New (and downstream) versions of the "virt" machine type may choose to reject VGA-like devices altogether. (I'm unsure how this would be implemented in practice however.)
This blog post from Gerd might contain information that are relevant to the issue at hand. https://www.kraxel.org/blog/2019/02/ramfb-display-in-qemu/
Support for the vfio ramfb option would be useful too, for vgpu boot display. Can be enabled today this way: <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'> [ ... ] </hostdev> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/> <qemu:arg value='-set'/> <qemu:arg value='device.hostdev0.ramfb=on'/> </qemu:commandline>
(In reply to Laszlo Ersek from comment #0) > Libvirt already supports the "virtio-gpu-pci" device for AARCH64 guests > (as the "virtio" model), but that's good enough only for Linux guests, > not for Windows guests. Windows requires a directly accessible > framebuffer, and virtio-gpu-pci doesn't provide one. So how do we go about testing this on windows / aarch64? > The issue also goes away with virtio-gpu-pci, because (a) there is no > framebuffer to speak of, (b) all display data, exchanged with virtio > transfers, are mapped as normal DRAM (WB) by both guest and host. > Unfortunately, Windows absolutely insist on a framebuffer for booting > (even though a direct access framebuffer is optional, according to the > UEFI spec). Therefore we can't use virtio-gpu-pci. > > (Again, don't confuse virtio-gpu-pci with virtio-vga. The latter is > virtio-gpu-pci PLUS the traditional VGA stuff. No good for aarch64.) > > The issue also goes away with the "ramfb" device model that Gerd wrote > for QEMU. It is a (comparatively) simple platform device that uses DRAM > (WB) as the framebuffer, so again, both sides' mappings agree, wrt. > cacheability. The UEFI driver (QemuRamfbDxe, also by Gerd) exposes the > direct-access framebuffer to the Windows boot loader, so the framebuffer > requirement is satisfied, while also side-stepping the non-coherence > issue on ARM64. Ramfb is not a PCI device, but it doesn't need to be. > UEFI-booted Windows is expected to work just with the UEFI framebuffer > that it inherits. Graphics mode changes and such are not expected to > work, of course, but basic display is expected to work. > > Please introduce the "ramfb" display device model in the domain XML. > Thanks! So, are you suggesting that this ramfb display device will be used as a primary display in these situations? In Gerd's blog post, he suggests that this is possible (via the ramfb standalone device) but not a good idea. Or are you suggesting that ramfb should be used in conjunction with virtio-gpu-pci somehow? In other words, if we add support in libvirt domain xml for ramfb, what is the expected configuration we'd like to support? and how should that translate to the qemu command? Gerd's suggestion in coment #4 seems to be a slightly separate issue.
> So, are you suggesting that this ramfb display device will be used as a > primary display in these situations? In Gerd's blog post, he suggests that > this is possible (via the ramfb standalone device) but not a good idea. It's not very efficient due to lack of dirty tracking. So, if you have something else (better), you should use that instead. Problem is, on aarch64 we don't have something else. So, yes, supporting ramfb makes sense. > Or > are you suggesting that ramfb should be used in conjunction with > virtio-gpu-pci somehow? That is another use case (you might want make that a separate bug). > In other words, if we add support in libvirt domain xml for ramfb, what is > the expected configuration we'd like to support? and how should that > translate to the qemu command? (1) standalone ramfb: <video> <model type='ramfb'/> Translates to '-device ramfb' (2) vgpu+ramfb <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on' ramfb='on'> Translates to '-device vfio-pci-nohotplug,ramfb=on,...'
Gerd replied in my place, better than I could have -- thanks. Clearing needinfo.
(In reply to Gerd Hoffmann from comment #6) Thanks Gerd, I thought that this was what was requested, but so far in my experiments, I have not been able to successfully use the standalone ramfb device as a primary display device. For example, early in the boot process, it shows a boot menu. But after booting the OS, the display does not update. What am I doing wrong here?
(In reply to Jonathon Jongsma from comment #8) > (In reply to Gerd Hoffmann from comment #6) > Thanks Gerd, I thought that this was what was requested, but so far in my > experiments, I have not been able to successfully use the standalone ramfb > device as a primary display device. For example, early in the boot process, > it shows a boot menu. But after booting the OS, the display does not update. > What am I doing wrong here? linux @ x86 bios I guess? You need vesafb in the guest then (vga=ask on the linux command line, then pick a vesa mode). vga text mode (which is used by vgacon) doesn't work because there is no vga device in the first place. Alternatively use something more modern: UEFI. edk2 comes with a ramfb driver and will setup a GOP on the ramfb device. Should work fine with all guests on both x86 and arm.
So, apparently I neglected to update this bug after implementing the feature. The ramfb display device model made it into upstream libvirt as of version 5.9.0. In addition, the vgpu boot display feature (see comment #4) made it into upstream 5.10.0. rhel-av 8.3 includes version 6.2.0 of libvirt, so this feature should already be implemented.
This feature was implemented in RHEL-AV 8.2.0.
Hi Gerd, When i tried to add 'ramfb‘ video device into guest xml, it reported error "unsupported configuration: domain configuration does not support 'video model' value 'ramfb'". Is there any other setting needed for 'ramfb' video device? #virsh edit vm1 ... <device> <video> <model type='ramfb'/> </video> ... </device> ... Also report the same error, if i added the qemu command line in the xml: #virsh edit vm1 <domain> ... <device> <video> <model type='ramfb'/> </video> ... </device> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/> <qemu:arg value='-set'/> <qemu:arg value='device.hostdev0.ramfb=on'/> </qemu:commandline> </domain>
Hmm, ramfb was patched out downstream (commit 67511676246cce57becbd2dcf5abccf08d9ef737, even though not mentioned in the commit message). So I guess we need a qemu-kvm bug to revert that.
Test with libvirt-6.0.0-24.module+el8.2.1+6997+c666f621.x86_64 and qemu-kvm-4.2.0-25.module+el8.2.1+6985+9fd9d514.x86_64. Test steps： 1.Start a guest with standalone 'ramfb' video device: #virsh start test1 # virsh dumpxml test1 | grep -A5 video <video> <model type='ramfb' heads='1' primary='yes'/> <alias name='video0'/> </video> Check the qemu cmd line: #ps aux | grep -i qemu-kvm ..-device ramfb,id=video0.. 2.For the display checking in guest os, please refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1841068#c15 - https://bugzilla.redhat.com/show_bug.cgi?id=1841068#c18
Move the bug to verified according to comment 14 - 22. And track 'vgpu+ramfb' in the bug 1847791.
Changing this TestOnly BZ as CLOSED CURRENTRELEASE. Please reopen if the issue is not resolved.