Description of problem: If use below xml snippet, <memoryBacking> <hugepages/> </memoryBacking> the corresponding qemu cli is: -mem-prealloc \ -mem-path /dev/hugepages/libvirt/qemu/1-rhel8 In qemu5.0, -mem-path and -mem-prealloc are aliased to '-M memory-backend' internally. So we request libvirt to make a change to use '-M memory-backend'. The usage of '-M memory-backend' is like this, # /usr/libexec/qemu-kvm \ -m 4G \ -object memory-backend-file,mem-path=/dev/hugepages/libvirt/qemu/1-rhel8 ,size=4G,id=mem0,prealloc=yes \ -M q35,memory-backend=mem0 Version-Release number of selected component (if applicable): libvirt-client-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64 qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Miso, was this already discussed upstream? Is it something we should prioritize? Thanks.
Writing the code is not a problem, but question is whether QEMU is able to migrate if the source uses the old command line and the destination would use the new command line. Also, is there any specific reason why the command line generated by libvirt is not good (I mean apart from aliasing)? I will need to check whether this new mode is somehow detectable through QMP so that libvirt knows whether QEMU supports the new mode or not.
(In reply to Michal Privoznik from comment #2) > Writing the code is not a problem, but question is whether QEMU is able to > migrate if the source uses the old command line and the destination would > use the new command line. No, migration would fail if source and destination use different command line. I guess it's in the same situation as https://www.redhat.com/archives/libvir-list/2020-May/msg00490.html. > Also, is there any specific reason why the command > line generated by libvirt is not good (I mean apart from aliasing)? I will leave this question to our QEMU developer Igor who introduced memory-backend property. Igor, would you please help explain? Thanks. > I will > need to check whether this new mode is somehow detectable through QMP so > that libvirt knows whether QEMU supports the new mode or not.
I'd like mem-prealloc and mem-path CLI to be deprecated and eventually removed, everything they are doing can be accomplished with new memory-backend property. As for migration, old and new way can be compatible, in case mem-path was used one should replace it with memory-backend-file object, and mem-prealloc is replaced by corresponding 'prealloc' property. One thing that's not available to libvirt is MachineClass::default_ram_id to keep migration stream compatible. 'id' isn't going to change ever (to keep migration working) so it could be hardcoded on libvirt side (typically each machine has its own id) or as an alternative, I can make a patch to expose it via query-machines. Another reason for supporting memory-backend is virtfs use-case (which requires file/fd backend) without using '-numa memdev' hack where numa is available and in-case of s390 (which lacks numa) memory-backend is the only way to use virtiofs.
(In reply to Igor Mammedov from comment #4) > I'd like mem-prealloc and mem-path CLI to be deprecated and eventually > removed, > everything they are doing can be accomplished with new memory-backend > property. > > As for migration, old and new way can be compatible, in case mem-path was > used > one should replace it with memory-backend-file object, and mem-prealloc is > replaced > by corresponding 'prealloc' property. One thing that's not available to > libvirt > is MachineClass::default_ram_id to keep migration stream compatible. Yeah, this is what I just hit in my testing. I wrote some patches and tested migration. If the id was random string (say "ram0"), then migration did not work. But if I changed it to "pc.ram" then it suddenly started working. I was able to migrate to even older QEMU which didn't have -machine memory-backend= support. Is this expected? Looking into gitlog, the name was introduced in v0.13.0-rc0~96 and kept unchanged (because as you say it would break migration). > > 'id' isn't going to change ever (to keep migration working) so it could be > hardcoded on libvirt side (typically each machine has its own id) or as > an alternative, I can make a patch to expose it via query-machines. Yeah, that could work. Libvirt can use it then to detect whether the -machine memory-backend= is supported or not. There is going to be one release of QEMU where it is supported and libvirt would still use the old cmd line, but I think that's acceptable. > > Another reason for supporting memory-backend is virtfs use-case (which > requires > file/fd backend) without using '-numa memdev' hack where numa is available > and in-case of s390 (which lacks numa) memory-backend is the only way to use > virtiofs. Yeah, my idea is to make the switch complete. I mean, not only use memory-backend-file instead of -mem-path but use memory-backend-ram for regular case. That simplifies implementation on libvirt side and possibly allows you to deprecate more stuff.
Patches proposed upstream: https://www.redhat.com/archives/libvir-list/2020-May/msg01114.html https://lists.nongnu.org/archive/html/qemu-devel/2020-05/msg07255.html
Rebased patches sent to the list (now that QEMU part is merged): https://www.redhat.com/archives/libvir-list/2020-September/msg00023.html
v2: https://www.redhat.com/archives/libvir-list/2020-September/msg00978.html
Merged upstream as: 88957116c9 qemu: Use memory-backend-* for regular guest memory b647654cbb qemu: Track default-ram-id machine attribute d1ffc8cd3e qemuBuildMemoryBackendProps: Fix const correctness a658a4bdf7 qemuBuildMemoryBackendProps: Prealloc mem for memfd backend 0217c5a6b4 qemuBuildMemoryBackendProps: Respect //memoryBacking/allocation/@mode=immediate eda5cc7a62 qemuBuildMemoryBackendProps: Move @prealloc setting to backend agnostic part v6.8.0-27-g88957116c9
We will need this patch too: https://www.redhat.com/archives/libvir-list/2020-October/msg00400.html
Pushed upstream as: 0c8ab47847 qemu: Don't generate '-machine memory-backend' and '-numa memdev' v6.8.0-232-g0c8ab47847
Moving to POST per comment 12.
Tested with libvirt-daemon-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 & qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64 S1: Start vm with hugepages and without numa node configuration in xml step: 1. the domain xml include below xml part - <memoryBacking> <hugepages> <page size='2048' unit='KiB'/> </hugepages> <source type='memfd'/> <allocation mode='immediate'/> </memoryBacking> and there is NO numa cell configuration. 2. Start vm and check the qemu-kvm cmdline <memoryBacking> <hugepages> <page size='2048' unit='KiB'/> </hugepages> <source type='memfd'/> <allocation mode='immediate'/> </memoryBacking> -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram \ -cpu IvyBridge-IBRS,ss=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-time,hv-vapic,hv-spinlocks=0x1000 \ -m 2560 \ -object memory-backend-memfd,id=pc.ram,hugetlb=yes,hugetlbsize=2097152,prealloc=yes,size=2684354560 \ Result: The memory-backend is used instead of "-mem-prealloc -mem-path" . s2: Migrate the vm to libvirt-daemon-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 & qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64 virsh migrate rhel8 qemu+ssh://10.73.xx.xx/system --live --verbose root.xx.xx's password: Migration: [100 %] s3: Migrate the vm with the same domain xml configuration from 8.3 to 8.4 libvirt-daemon-6.6.0-9.module+el8.3.1+9131+fb7f8c9f.x86_64 & qemu-kvm-5.1.0-16.module+el8.3.1+8958+410ab178.x86_64 # virsh migrate rhel8 qemu+ssh://10.73.xx.xx/system --live --verbose --unsafe root.xx.xx's password: error: internal error: qemu unexpectedly closed the monitor: 2020-12-14T09:05:25.349045Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead 2020-12-14T09:05:25.669997Z qemu-kvm: Unknown ramblock "pc.ram", cannot accept migration 2020-12-14T09:05:25.670041Z qemu-kvm: error while loading state for instance 0x0 of device 'ram' 2020-12-14T09:05:25.670457Z qemu-kvm: load of migration failed: Invalid argument Michal, could you please help to confirm if it's accepted for s3, Thanks!
I think we need Igor's opinion.
(In reply to Jing Qi from comment #14) > Tested with libvirt-daemon-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 & > qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64 > > S1: Start vm with hugepages and without numa node configuration in xml > step: > 1. the domain xml include below xml part - > <memoryBacking> > <hugepages> > <page size='2048' unit='KiB'/> > </hugepages> > <source type='memfd'/> > <allocation mode='immediate'/> > </memoryBacking> > > > and there is NO numa cell configuration. > > 2. Start vm and check the qemu-kvm cmdline > > <memoryBacking> > <hugepages> > <page size='2048' unit='KiB'/> > </hugepages> > <source type='memfd'/> > <allocation mode='immediate'/> > </memoryBacking> > > -machine > pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc. > ram \ > -cpu > IvyBridge-IBRS,ss=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md- > clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on, > ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv- > time,hv-vapic,hv-spinlocks=0x1000 \ > -m 2560 \ > -object > memory-backend-memfd,id=pc.ram,hugetlb=yes,hugetlbsize=2097152,prealloc=yes, > size=2684354560 \ > [...] > > s3: Migrate the vm with the same domain xml configuration from 8.3 to 8.4 > libvirt-daemon-6.6.0-9.module+el8.3.1+9131+fb7f8c9f.x86_64 & > qemu-kvm-5.1.0-16.module+el8.3.1+8958+410ab178.x86_64 > > # virsh migrate rhel8 qemu+ssh://10.73.xx.xx/system --live --verbose --unsafe > root.xx.xx's password: > error: internal error: qemu unexpectedly closed the monitor: > 2020-12-14T09:05:25.349045Z qemu-kvm: -device > cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is > deprecated, please use a different VGA card instead > 2020-12-14T09:05:25.669997Z qemu-kvm: Unknown ramblock "pc.ram", cannot > accept migration > 2020-12-14T09:05:25.670041Z qemu-kvm: error while loading state for instance > 0x0 of device 'ram' > 2020-12-14T09:05:25.670457Z qemu-kvm: load of migration failed: Invalid > argument > > Michal, could you please help to confirm if it's accepted for s3, Thanks! I'd say it should work, what was target qemu CLI in this case?
I've just did some quick check and I was able to migrate current libvirt and qemu with default RAM id and without it. I mean, back and forth between: -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram \ -cpu qemu64 \ -m 4096 \ -object memory-backend-ram,id=pc.ram,size=4294967296 \ and -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off \ -cpu qemu64,x2apic=on,hypervisor=on,lahf-lm=on,svm=off \ -m 4096 \ libvirt: v6.10.0-206-gbff2ad5d6b qemu: v5.2.0-329-g17584289af Jing, can you share your command lines from the source and the destination?
Created attachment 1739218 [details] source qemu command file
Created attachment 1739219 [details] target qemu command file
I used "pc-i440fx-rhel7.6.0" for the testing since there is a virtio-blk issue currently for "q35" machine. And the target and source qemu command files were uploaded.
Ah, forgot about the memfd. I've used regular pages. Anyway, retested and these are the interesting bits from source and destination: -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram \ -cpu qemu64 \ -m 4096 \ -object memory-backend-memfd,id=pc.ram,hugetlb=yes,hugetlbsize=2097152,prealloc=yes,size=4294967296 \ -machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off \ -cpu qemu64,x2apic=on,hypervisor=on,lahf-lm=on,svm=off \ -m 4096 \ -mem-prealloc \ -mem-path /hugepages2M/libvirt/qemu/4-fedora30 \ (yes, I know on the destination libvirt did not use memfd because of bug 1839034) But anyway, I was able to migrate there and back successfully. Igor, got any clues?
Jing, does adding 'x-use-canonical-path-for-ramblock-id=off' option to backend on target helps?
Igor, Yes. With adding 'x-use-canonical-path-for-ramblock-id=off', the migration can succeed. source qemu cmd: /usr/libexec/qemu-kvm -name guest=rhel8,debug-threads=on -S -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu IvyBridge-IBRS,ss=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-time,hv-vapic,hv-spinlocks=0x1000 -m 2560 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu/1-rhel8 -overcommit mem-lock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 60d7e03d-0d7b-42c6-ac4c-4da98dd6a5ec -display none -no-user-config -nodefaults -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -blockdev '{"driver":"file","filename":"/nfs/RHEL-8.4-x86_64-latest.qcow2","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' -device ide-hd,bus=ide.0,unit=0,drive=libvirt-1-format,id=ide0-0-0,bootindex=1,write-cache=on -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -monitor stdio target qemu cmd: /usr/libexec/qemu-kvm -name guest=rhel8,debug-threads=on -S -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram -cpu IvyBridge-IBRS,ss=on,pcid=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-time,hv-vapic,hv-spinlocks=0x1000 -m 2560 -object memory-backend-file,id=pc.ram,mem-path=/dev/hugepages/libvirt/qemu/10-rhel8,prealloc=yes,size=2684354560,x-use-canonical-path-for-ramblock-id=off -overcommit mem-lock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 60d7e03d-0d7b-42c6-ac4c-4da98dd6a5ec -display none -no-user-config -nodefaults -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -blockdev '{"driver":"file","filename":"/nfs/RHEL-8.4-x86_64-latest.qcow2","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' -device ide-hd,bus=ide.0,unit=0,drive=libvirt-1-format,id=ide0-0-0,bootindex=1,write-cache=on -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on -incoming defer -monitor stdio
Looks like we broke compat machinery somehow when moved to qemu-5.2 CCing David. PS: issue needs it's own BZ
(In reply to Igor Mammedov from comment #26) > Looks like we broke compat machinery somehow when moved to qemu-5.2 > CCing David. > > PS: > issue needs it's own BZ Filed new bug 1912201 for it .
Duplicating comment https://bugzilla.redhat.com/show_bug.cgi?id=1912201#c10 here" Issue is that when -machine memory-backend=pc.ram \ -object memory-backend-memfd,id=pc.ram created backend may change MemoryRegion name to canonical one that adds '/object/' to 'id', when compat property is set to 'x-use-canonical-path-for-ramblock-id=on'. so for old machine types where 'x-use-canonical-path-for-ramblock-id=on', -object creates backend with MemoryRegion 'id' that contains '/object/' in it, hen migration fails as old target QEMU or QEMU started without -machine memory-backend expect prefix-less 'id' in migration stream. It's possible to fix on libvirt side by always adding 'x-use-canonical-path-for-ramblock-id=off' to to the backend that will be used with -machine memory-backend. (That's what legacy -m X, does under hood, note x-use-canonical-path-for-ramblock-id option is available since qemu-4.0) Another alternative could be not using '-machine memory-backend' for machines older than upstream 4.0 and downstream rhel8.0.
Basically if QEMU supports '-machine memory-backend', then always feed it backend with 'x-use-canonical-path-for-ramblock-id=off' option.
(In reply to Igor Mammedov from comment #31) > Basically if QEMU supports '-machine memory-backend', > then always feed it backend with 'x-use-canonical-path-for-ramblock-id=off' > option. The way I read qemu code it's necessary to set only if memory-backend-memfd/memory-backend-file is a replacement for '-m X', right? For other cases, it should not be set, right? For instance NUMA: -numa nome,memdev=$ID -object memory-backend-memfd,id=$ID. If it is so, then it should be fairly simple to implement. But it feels a bit hackish.
(In reply to Michal Privoznik from comment #32) > (In reply to Igor Mammedov from comment #31) > > Basically if QEMU supports '-machine memory-backend', > > then always feed it backend with 'x-use-canonical-path-for-ramblock-id=off' > > option. > > The way I read qemu code it's necessary to set only if > memory-backend-memfd/memory-backend-file is a replacement for '-m X', right? yes, only for a backend that will be used with '-machine memory-backend' reason for that is that MemoryRegion backend is replacing doesn't use any prefix. > For other cases, it should not be set, right? For instance NUMA: -numa > nome,memdev=$ID -object memory-backend-memfd,id=$ID. > > If it is so, then it should be fairly simple to implement. But it feels a > bit hackish. it's looks like that, but there is literally nothing we can do about it on QEMU side, since backend is provided by user. option is necessary only for keeping migration working for old machine types (before 4.0/rhel76). For newer machine types it's no necessary since it's 'off' by default. I'll post a patch to update '-machine memory-backend' doc comment, so it would mention this caveat.
If libvirt is required to use a property with the 'x-' prefix (As I understand there isn't any other option to keep compatibility with older versions otherwise) we'll need some form of guarantee that the 'x-' prefixed parameter will not be modified in the future. By guarantee I mean adding documentation and code comment which state that this is considered stable regardless of the 'x-' policy used for other properties. That way we'll be able to waive the requirement that no 'x-' prefixed stuff is used in libvirt.
(In reply to Peter Krempa from comment #34) > If libvirt is required to use a property with the 'x-' prefix (As I > understand there isn't any other option to keep compatibility with older > versions otherwise) we'll need some form of guarantee that the 'x-' prefixed > parameter will not be modified in the future. By guarantee I mean adding > documentation and code comment which state that this is considered stable > regardless of the 'x-' policy used for other properties. That way we'll be > able to waive the requirement that no 'x-' prefixed stuff is used in libvirt. sure, I'll post a patch. Michal, x-use-canonical-path-for-ramblock-id is only applicable to memory-backend-memfd/memory-backend-file.
Since we still have a serious blocker (bug 1912201) I'm moving this back to ASSIGNED. The feature is unusable until the blocker is fixed.
The problem mentioned in comment 36 was fixed upstream. Moving back to ON_QA.
Verified with version - libvirt-7.0.0-4.module+el8.4.0+10093+e085f1eb.x86_64 & qemu-kvm-5.2.0-5.module+el8.4.0+9775+0937c167.x86_64 The test steps are included in comment 14 above (S1 & S2) and comment 24 (S3) in bug 1912201
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098