Hi Yalan,
I'm curious how could you configure shared memory without hugepage. 
Can you provide the full qemu command under below two scenarios so that I could have a try from qemu side:
1) with shared memory and hugepage;
2) with shared memory but no hugepage setting.

(In reply to Li Xiaohui from comment #1)
> Hi Yalan,
> I'm curious how could you configure shared memory without hugepage. 
Yes, libvirt and qemu do not report any error when I start the vm.

> Can you provide the full qemu command under below two scenarios so that I
> could have a try from qemu side:
> 1) with shared memory and hugepage;

qemu      147772 99.4  0.2 6280344 75612 ?       Sl   02:32   0:12 /usr/libexec/qemu-kvm -name guest=rhel
......
 -machine pc-q35-rhel9.0.0,usb=off,dump-guest-core=off ……
-m 2048 -overcommit mem-lock=off -smp 8,sockets=8,cores=1,threads=1 
-object {"qom-type":"memory-backend-file","id":"ram-node0","mem-path":"/dev/hugepages/libvirt/qemu/17-rhel","share":true,"prealloc":true,"size":2147483648} 
-numa node,nodeid=0,cpus=0-7,memdev=ram-node0 
-uuid 7ffd0ecf-504b-43a9-91f4-f339e98d136b -no-user-config -nodefaults 


> 2) with shared memory but no hugepage setting.

qemu      145371 39.8  1.8 7234220 596756 ?      Sl   02:04   0:26 /usr/libexec/qemu-kvm -name guest=rhel,debug-threads=on -S -object 
……
-machine pc-q35-rhel9.0.0,usb=off,dump-guest-core=off -accel kvm 
……
-m 2048 -overcommit mem-lock=off -smp 8,sockets=8,cores=1,threads=1 
-object {"qom-type":"memory-backend-file","id":"ram-node0","mem-path":"/var/lib/libvirt/qemu/ram/17-rhel/ram-node0","share":true,"size":2147483648} 
-numa node,nodeid=0,cpus=0-7,memdev=ram-node0 -uuid 7ffd0ecf-504b-43a9-91f4-f339e98d136b -no-user-config -nodefaults

Thanks Yalan for providing qemu commands.

Reproduce bug both through qemu and libvirt on rhel9.0.0 (qemu-kvm-6.2.0-10.el9.x86_64 & libvirt-8.0.0-5.el9.x86_64):
1.If we set hugepage on src and dst host, then postcopy migration succeed;
2.If we don't set hugepage and don't mount the mem-path hugetlbfs on hosts, then postcopy will fail, migration on src host would pause, and qemu on dst host would quit by error. But plain migration succeed under this scenario.

Reproduce step:
1.Create a directory on src and dst host:
# /var/lib/libvirt/qemu/ram/17-rhel
2.Boot vm with qemu cmds:
-machine q35 \
-m 4096 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
......
-object memory-backend-file,id=ram-node0,mem-path=/var/lib/libvirt/qemu/ram/17-rhel/ram-node0,share=true,size=4096M \
-numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
3.Migrate vm from src to dst host through postcopy


Actual result:
Migration would pause during postcopy active stage.

(1)src qemu:
(qemu) 2022-03-01T03:56:48.244543Z qemu-kvm: failed to save SaveStateEntry with id(name): 1(ram)
2022-03-01T03:56:48.244598Z qemu-kvm: Detected IO failure for postcopy. Migration paused.
(2)dst qemu:
(qemu) 2022-03-01T03:56:40.085671Z qemu-kvm: ram_block_enable_notify userfault register: Invalid argument
2022-03-01T03:56:40.085761Z qemu-kvm: ram_block_enable_notify failed
2022-03-01T03:56:40.085943Z qemu-kvm: cleanup_range: userfault unregister Invalid argument
2022-03-01T03:56:48.242290Z qemu-kvm: load of migration failed: Operation not permitted

And if postcopy with memory-backend-ram, it also succeeds:
-machine q35 \
-m 4096 \
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
...
-object memory-backend-ram,id=ram-node0,share=true,size=4096M \
-numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \

Hi David, can you check Comment 3, whether we support boot vm with memory-backend-file but none hugepage configured? And shall we give some error prompt when we start vm under such scenario?

(In reply to Li Xiaohui from comment #3)
> Thanks Yalan for providing qemu commands.
> 
> Reproduce bug both through qemu and libvirt on rhel9.0.0
> (qemu-kvm-6.2.0-10.el9.x86_64 & libvirt-8.0.0-5.el9.x86_64):
> 1.If we set hugepage on src and dst host, then postcopy migration succeed;
> 2.If we don't set hugepage and don't mount the mem-path hugetlbfs on hosts,
> then postcopy will fail, migration on src host would pause, and qemu on dst
> host would quit by error. But plain migration succeed under this scenario.
> 
> Reproduce step:
> 1.Create a directory on src and dst host:
> # /var/lib/libvirt/qemu/ram/17-rhel
> 2.Boot vm with qemu cmds:
> -machine q35 \
> -m 4096 \
> -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
> ......
> -object
> memory-backend-file,id=ram-node0,mem-path=/var/lib/libvirt/qemu/ram/17-rhel/
> ram-node0,share=true,size=4096M \
> -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
> 3.Migrate vm from src to dst host through postcopy
> 
> 
> Actual result:
> Migration would pause during postcopy active stage.
> 
> (1)src qemu:
> (qemu) 2022-03-01T03:56:48.244543Z qemu-kvm: failed to save SaveStateEntry
> with id(name): 1(ram)
> 2022-03-01T03:56:48.244598Z qemu-kvm: Detected IO failure for postcopy.
> Migration paused.
> (2)dst qemu:
> (qemu) 2022-03-01T03:56:40.085671Z qemu-kvm: ram_block_enable_notify
> userfault register: Invalid argument
> 2022-03-01T03:56:40.085761Z qemu-kvm: ram_block_enable_notify failed
> 2022-03-01T03:56:40.085943Z qemu-kvm: cleanup_range: userfault unregister
> Invalid argument
> 2022-03-01T03:56:48.242290Z qemu-kvm: load of migration failed: Operation
> not permitted


That's an invalid configuration for postcopy; using '/var/lib/libvirt/qemu/ram/17-rhel' only works if it's actually
a shared memory filesystem, not if it's a normal on disc file.

So please retest using a directory on /dev/shm or a separate shmfs mount.

(In reply to Li Xiaohui from comment #5)
> Hi David, can you check Comment 3, whether we support boot vm with
> memory-backend-file but none hugepage configured? And shall we give some
> error prompt when we start vm under such scenario?

We do support postcopy with memory-backend-file in case it references shared memory ("virtual file" on shmem). We don't support postcopy with memory-backend-file in case it references an actual file.

I agree that the error message is somehwat suboptimal: it merely states that usefaultfd (use by postcopy) is not supported on that memory backing.

(In reply to Dr. David Alan Gilbert from comment #6) 
> That's an invalid configuration for postcopy; using
> '/var/lib/libvirt/qemu/ram/17-rhel' only works if it's actually
> a shared memory filesystem, not if it's a normal on disc file.
> 
> So please retest using a directory on /dev/shm or a separate shmfs mount.

It works well that vm migration with postcopy and shared memory on /dev/shm.

Steps:
1. On source and target host, update the memory backing in qemu.conf:
# grep -v ^# /etc/libvirt/qemu.conf | grep -v '^$'
memory_backing_dir = "/dev/shm"

Restart libvirt related services, and prepare the migration env.

2. Start a vm:
# virsh dumpxml rhel
......
<memoryBacking>
    <source type='file'/>
    <access mode='shared'/>
  </memoryBacking>
......
# virsh start rhel 
Domain 'rhel' started

Qemu command line:
-machine pc-q35-rhel9.0.0,usb=off,dump-guest-core=off,memory-backend=pc.ram 
-accel kvm 
-cpu Haswell-noTSX-IBRS 
-m 1024 
-object {"qom-type":"memory-backend-file","id":"pc.ram","mem-path":"/dev/shm/libvirt/qemu/4-rhel/pc.ram","share":true,"x-use-canonical-path-for-ramblock-id":false,"size":1073741824} 
-overcommit mem-lock=off

3. Migration succeed:
# virsh migrate rhel  qemu+ssh://dell-per730-36.lab.eng.pek2.redhat.com/system --live --verbose --postcopy --timeout 5 --timeout-postcopy   --bandwidth 5  --p2p   --persistent --undefinesource 
Migration: [100 %]

Per Comment 6 and Comment 7 and Comment 8, shall we close this bug as NOTABUG? 
Or optimize our error prompts and block postcopy migration before it starting without a shared memory filesystem?

And I have another question: 
As it is confirmed that **"mem-path":"/var/lib/libvirt/qemu/ram/17-rhel/ram-node0","share":true"** is a invalid setting and VM can start with it successfully, what could happen for VMs with such setting? the memory can not be shared, right? If that't true, maybe we should forbid to start the VM, otherwise it will be misused, for example, with virtiofs or vhostuser interface which need shared memory to work properly.

IIUC sharing is not the problem, however..  Do/Should we support running VM with memory allocated on a generic file system at all?  Do we have any use case of that?

(In reply to Peter Xu from comment #11)
> IIUC sharing is not the problem, however..  Do/Should we support running VM
> with memory allocated on a generic file system at all?  Do we have any use
> case of that?

IIRC yes we support that. The use case is essentially having large VMs and using the pagecache as a replacement for swapping. I remember someone (Dave G. ?) mention that we do have customer installations making use of that. Of course, they are not using postcopy, because it cannot possibly work right now.

(In reply to yalzhang from comment #10)
> And I have another question: 
> As it is confirmed that
> **"mem-path":"/var/lib/libvirt/qemu/ram/17-rhel/ram-node0","share":true"**
> is a invalid setting and VM can start with it successfully, what could
> happen for VMs with such setting? the memory can not be shared, right? If
> that't true, maybe we should forbid to start the VM, otherwise it will be
> misused, for example, with virtiofs or vhostuser interface which need shared
> memory to work properly.

IIRC vhost-user should work just fine with shared file mappings. After all, the only requirement they have is communicating with the VM via the guest RAM being mapped into their address space via via mmap(MAP_SHARED). And there are valid use cases for that.

The only limitation is that userfaultfd only supports shmem, not *any* shared memory.

(In reply to Li Xiaohui from comment #9)
> Per Comment 6 and Comment 7 and Comment 8, shall we close this bug as
> NOTABUG? 
> Or optimize our error prompts and block postcopy migration before it
> starting without a shared memory filesystem?

While I think it is NOTABUG, and nothing was actually harmed (the VM is paused just like if migration failed IIRC),.

However, an error message that at least says something like "userfaultfd could not be initialized, postcopy is not supported." would be preferable.

(In reply to David Hildenbrand from comment #13)
> (In reply to yalzhang from comment #10)
> > And I have another question: 
> > As it is confirmed that
> > **"mem-path":"/var/lib/libvirt/qemu/ram/17-rhel/ram-node0","share":true"**
> > is a invalid setting and VM can start with it successfully, what could
> > happen for VMs with such setting? the memory can not be shared, right? If
> > that't true, maybe we should forbid to start the VM, otherwise it will be
> > misused, for example, with virtiofs or vhostuser interface which need shared
> > memory to work properly.
> 
> IIRC vhost-user should work just fine with shared file mappings. After all,
> the only requirement they have is communicating with the VM via the guest
> RAM being mapped into their address space via via mmap(MAP_SHARED). And
> there are valid use cases for that.

What's the valid use case?

> 
> The only limitation is that userfaultfd only supports shmem, not *any*
> shared memory.

(In reply to Dr. David Alan Gilbert from comment #15)
> (In reply to David Hildenbrand from comment #13)
> > (In reply to yalzhang from comment #10)
> > > And I have another question: 
> > > As it is confirmed that
> > > **"mem-path":"/var/lib/libvirt/qemu/ram/17-rhel/ram-node0","share":true"**
> > > is a invalid setting and VM can start with it successfully, what could
> > > happen for VMs with such setting? the memory can not be shared, right? If
> > > that't true, maybe we should forbid to start the VM, otherwise it will be
> > > misused, for example, with virtiofs or vhostuser interface which need shared
> > > memory to work properly.
> > 
> > IIRC vhost-user should work just fine with shared file mappings. After all,
> > the only requirement they have is communicating with the VM via the guest
> > RAM being mapped into their address space via via mmap(MAP_SHARED). And
> > there are valid use cases for that.
> 
> What's the valid use case?

I was told that we have some customers that run large VMs on almost swap-less systems. Instead, you use the file as a replacement for the swap, but more fine-grained, targeting only VM RAM. (not sure if there are also use cases for offline-migration of VMs via network storage)

What happens when the memory is swapped of for migration?

I.e. we have a page, that is swapped to the file, and then we migrate?  To finish migration, we need to re-read the file back to memory, to send it, and we can found that we don't have space on the page cache.  I am not sure that we really want to support migration with non-shared filesystems.

Setting priority & severity to 'medium' based on:
- According to comments, seems like a NOTABUG, but may require better error message
- There is an ongoing conversation at the topic, but no updates for a while (6mo+)

Dave, Juan, Peter, David Hildenbrand. Is that ok?
Please change it if you see fit.

(In reply to Juan Quintela from comment #17)
> What happens when the memory is swapped of for migration?

[managed to ignore this BZ, somehow]

Just that same as happens with ordinary anonymous memory with swap.

> 
> I.e. we have a page, that is swapped to the file, and then we migrate?  To
> finish migration, we need to re-read the file back to memory, to send it,
> and we can found that we don't have space on the page cache.  I am not sure
> that we really want to support migration with non-shared filesystems.

Again, just the same as happens when we have !shared guest memory swapped out.

If the kernel is running out of memory, it has to reclaim memory (e.g., swapout other anon pages, evict other pagecache pages) such that user space major faults can be served.

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Test two scenarios from qemu side (qemu-kvm-8.0.0-4.el9.x86_64)
1. postcopy migration without a shared memory filesystem
(1) enable postcopy on src and dst host: the postcopy can be enabled on src host, but fail to enable on dst host, get prompt:
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"postcopy-ram","state":true}]}}
{"error": {"class": "GenericError", "desc": "Postcopy is not supported: Host backend files need to be TMPFS or HUGETLBFS only"}}
(2) try to start migration under postcopy enabled on src host but disable on dst host, would get migration failure on src host, and the qemu process would quit by automatically:
a. src host:
(qemu) migrate -d tcp:$dst_host_ip:1234
(qemu) 2023-05-30T06:49:37.280422Z qemu-kvm: Unable to write to socket: Bad file descriptor
(qemu) info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: failed (Unable to write to socket: Bad file descriptor)
total time: 0 ms

b: dst host:
(qemu) 2023-05-30T06:49:37.225877Z qemu-kvm: RAM postcopy is disabled but have 16 byte advise
2023-05-30T06:49:37.226528Z qemu-kvm: load of migration failed: Invalid argument

2. postcopy migration with shared memory filesystem, postcopy succeeds


Also tried above two scenarios through libvirt (libvirt-9.3.0-2.el9.x86_64)
1. Libvirt would report a error if try to do postcopy without shared filesystem
[root@dell-per7525-25 home]# virsh migrate rhel930 --live --verbose qemu+ssh://10.73.2.82/system --p2p   --persistent --undefinesource --postcopy --timeout 5 --timeout-postcopy
error: internal error: unable to execute QEMU command 'migrate-set-capabilities': Postcopy is not supported: Host backend files need to be TMPFS or HUGETLBFS only

2. Libvirt succeeds to do postcopy migration with shared filesystem
[root@dell-per7525-25 home]# virsh migrate rhel930 --live --verbose qemu+ssh://10.73.2.82/system --p2p   --persistent --undefinesource --postcopy --timeout 5 --timeout-postcopy
Migration: [100 %]

Note: 
(1) On source and target host, update the memory backing in qemu.conf:
# grep -v ^# /etc/libvirt/qemu.conf | grep -v '^$'
memory_backing_dir = "/dev/shm"

Restart libvirt related services, and prepare the migration env.

(2) Start a vm with memoryBacking configure, so virsh edit the domain:
# virsh dumpxml rhel9.3.0
......
<memoryBacking>
    <source type='file'/>
    <access mode='shared'/>
  </memoryBacking>

(In reply to Li Xiaohui from comment #29)
> Test two scenarios from qemu side (qemu-kvm-8.0.0-4.el9.x86_64)
> 1. postcopy migration without a shared memory filesystem
> (1) enable postcopy on src and dst host: the postcopy can be enabled on src
> host, but fail to enable on dst host, get prompt:
> {"execute":"migrate-set-capabilities","arguments":{"capabilities":
> [{"capability":"postcopy-ram","state":true}]}}
> {"error": {"class": "GenericError", "desc": "Postcopy is not supported: Host
> backend files need to be TMPFS or HUGETLBFS only"}}
> (2) try to start migration under postcopy enabled on src host but disable on
> dst host, would get migration failure on src host, and the qemu process
> would quit by automatically:

I mean the qemu process of dst host would quit by automatically.

> a. src host:
> (qemu) migrate -d tcp:$dst_host_ip:1234
> (qemu) 2023-05-30T06:49:37.280422Z qemu-kvm: Unable to write to socket: Bad
> file descriptor
> (qemu) info migrate
> globals:
> store-global-state: on
> only-migratable: off
> send-configuration: on
> send-section-footer: on
> decompress-error-check: on
> clear-bitmap-shift: 18
> Migration status: failed (Unable to write to socket: Bad file descriptor)
> total time: 0 ms
> 
> b: dst host:
> (qemu) 2023-05-30T06:49:37.225877Z qemu-kvm: RAM postcopy is disabled but
> have 16 byte advise
> 2023-05-30T06:49:37.226528Z qemu-kvm: load of migration failed: Invalid
> argument

Hi Peter,
I found one new issue when test scenario 1 according to Comment 29, shall we file a bug for it?
> 1. postcopy migration without a shared memory filesystem
> (1) enable postcopy on src and dst host: the postcopy can be enabled on src host, but fail to enable on dst host, get prompt:
> {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"postcopy-ram","state":true}]}}
> {"error": {"class": "GenericError", "desc": "Postcopy is not supported: Host backend files need to be TMPFS or HUGETLBFS only"}}
> (2) try to start migration under postcopy enabled on src host but disable on dst host, would get migration failure on src host, and > the qemu process would quit by automatically: 

after (2) step, I continue to do live migration without postcopy enabled:
(3) I would restart a VM with '-incoming defer' on dst host.
(4) disable postcopy capabilities on src host
(5) start to live migration

Actual result:
After step 5, migration would complete, but the qemu on dst host would quit automatically with the below error:
(qemu) 2023-05-30T09:49:59.252257Z qemu-kvm: error while loading state section id 1(ram)
2023-05-30T09:50:15.935819Z qemu-kvm: load of migration failed: Input/output error


BTW, libvirt works well when try to do live migration after postcopy failure without shared filesystem:
[root@dell-per7525-25 home]# virsh migrate rhel930 --live --verbose qemu+ssh://10.73.2.82/system --p2p   --persistent --undefinesource --postcopy --timeout 5 --timeout-postcopy
error: internal error: unable to execute QEMU command 'migrate-set-capabilities': Postcopy is not supported: Host backend files need to be TMPFS or HUGETLBFS only

[root@dell-per7525-25 home]# virsh migrate rhel930 --live --verbose qemu+ssh://10.73.2.82/system --p2p   --persistent --undefinesource
Migration: [100 %]

(In reply to Li Xiaohui from comment #29)
> (2) try to start migration under postcopy enabled on src host but disable on
> dst host, would get migration failure on src host, and the qemu process
> would quit by automatically:

Xiaohui,

Let's avoid testing use cases where we apply different migration capabilities on src & dst.

The thing is we always assume both qemus will apply the same capabilities or it's already wrong usage (maybe in the future we can teach migration framework to do handshakes, so they can communicate on the features, but not for now).  Just like when we specify different devices/cmdlines for src/dst and migration can easily go wrong.  We just so far rely on this to make everything work.

Thanks,
Peter

(In reply to Peter Xu from comment #32)
> (In reply to Li Xiaohui from comment #29)
> > (2) try to start migration under postcopy enabled on src host but disable on
> > dst host, would get migration failure on src host, and the qemu process
> > would quit by automatically:
> 
> Xiaohui,
> 
> Let's avoid testing use cases where we apply different migration
> capabilities on src & dst.
> 
> The thing is we always assume both qemus will apply the same capabilities or
> it's already wrong usage (maybe in the future we can teach migration
> framework to do handshakes, so they can communicate on the features, but not
> for now).  Just like when we specify different devices/cmdlines for src/dst
> and migration can easily go wrong.  We just so far rely on this to make
> everything work.

I see, thanks for the explanation.

I also tried again on the useful scenario 1 -> postcopy migration without a shared memory filesystem
(1) try to enable postcopy on src and dst host, after finding postcopy can't be enabled on dst host, then disable postcopy on src host;
(2) try to migrate VM from src to dst, migration succeeds, VM works well on dst host.



Per above test results, mark this bug verfied.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368