Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1870048

Summary: virtio-fs: failed to boot up one vm if try to boot up two vms simultaneously to share the same directory on the host
Product: Red Hat Enterprise Linux 9 Reporter: xiagao
Component: qemu-kvmAssignee: Hanna Czenczek <hreitz>
qemu-kvm sub component: virtio-fs QA Contact: xiagao
Status: CLOSED NOTABUG Docs Contact:
Severity: medium    
Priority: medium CC: dgilbert, drjones, kkiwi, lijin, virt-maint, xiagao
Version: unspecifiedKeywords: Regression, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-19 08:35:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1897025    
Attachments:
Description Flags
Launch two VMs simultaneously, and mount the same host directory on both through virtio-fs none

Description xiagao 2020-08-19 09:30:41 UTC 
Description of problem:
Boot up two guests to access the same directory tree on the host, if boot up them almost simultaneously, one of them will fail to boot up.

Version-Release number of selected component (if applicable):
qemu-kvm-5.1.0-2.module+el8.3.0+7652+b30e6901.x86_64
kernel-4.18.0-232.el8.x86_64

guest kernel:
kernel-4.18.0-232.el8.x86_64


How reproducible:
4/10

Steps to Reproduce:
1.create a shared dir
# mkdir /tmp/virtio_fs_test

2.Run the 1st virtiofsd daemon on the host. 
# /usr/libexec/virtiofsd --socket-path=/tmp/vhostqemu1 -o source=/tmp/virtio_fs_test -d

3.Boot the 1st vm with virtio fs device.
-object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm111,share=on \
-numa node,memdev=mem \
-chardev socket,id=char0,path=/tmp/vhostqemu1 \
-device vhost-user-fs-pci,chardev=char0,tag=myfs123,bus=pci.7,queue-size=1024 \

4.Run the 2nd virtiofsd daemon on the host，the interval of 
# /usr/libexec/virtiofsd --socket-path=/tmp/vhostqemu2 -o source=/tmp/virtio_fs_test -d

5.Boot the 2nd vm with virtio fs device
-object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm222,share=on \
-numa node,memdev=mem \
-chardev socket,id=char0,path=/tmp/vhostqemu2 \
-device vhost-user-fs-pci,chardev=char0,tag=myfs123,bus=pci.7,queue-size=1024 \


Actual results:
One of the vm will fail to start.

Expected results:
Both of vms can boot up simultaneously.

Additional info:
Didn't hit this issue on rhel8.3.0 slow train
qemu version:
qemu-kvm-4.2.0-32.module+el8.3.0+7629+c86ce105.x86_64
Comment 1 xiagao 2020-08-20 01:24:28 UTC 
Run 80 times with qemu-kvm-4.2.0-29.module+el8.2.1+7668+93a32c6b.1.x86_64, all pass.
So it's a regression.
Comment 2 xiagao 2020-09-23 02:15:42 UTC 
Hit a similar issue with virtio-fs device.
1) boot up vm with file-backed memory.
virtiofsd daemon:
#  /usr/libexec/virtiofsd --socket-path=/tmp/vhostqemu1 -o source=/home/virtiofs_test1 -o cache=always -d
qemu-cmd line:
-m 4G -smp 4,maxcpus=12,cores=4,threads=1,sockets=3
-chardev socket,id=char0,path=/tmp/vhostqemu1 \
-device vhost-user-fs-pci,chardev=char0,tag=myfs_11,bus=pci.11 \
-object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm1,share=on \
-numa node,memdev=mem \

2）shutdown this vm and start it again with step1, but it failed.
(qemu) qemu-kvm: -device ide-drive,bus=ide.0,unit=0,drive=drive_system_disk,id=ide0-0-0: warning: 'ide-drive' is deprecated, please use 'ide-hd' or 'ide-cd' instead
qemu-kvm: -device ide-drive,drive=drive-ide0-1-0,id=ide0-1-0,serial=GAGENT_TEST: warning: 'ide-drive' is deprecated, please use 'ide-hd' or 'ide-cd' instead
error: kvm run failed Bad address
EAX=00000000 EBX=00000000 ECX=00000010 EDX=000efced
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00007000
EIP=000efced EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000f6200 00000037
IDT=     000f623e 00000000
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=47 d0 0f 00 0f b7 d2 8d 44 24 02 e8 46 c4 ff ff 83 c4 28 c3 <53> 83 ec 20 e8 38 c8 ff ff bb 00 00 00 40 8d 44 24 0c 50 8d 44 24 0c 50 8d 4c 24 0c 8d 54

3)change mem-path to another dir, vm still couldn't boot up.
-object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm2,share=on \
Comment 3 xiagao 2020-09-23 02:23:37 UTC 
qemu-kvm-5.1.0-7.module+el8.3.0+8099+dba2fe3e.x86_64(In reply to xiagao from comment #2)
> Hit a similar issue with virtio-fs device.
> 1) boot up vm with file-backed memory.
> virtiofsd daemon:
> #  /usr/libexec/virtiofsd --socket-path=/tmp/vhostqemu1 -o
> source=/home/virtiofs_test1 -o cache=always -d
> qemu-cmd line:
> -m 4G -smp 4,maxcpus=12,cores=4,threads=1,sockets=3
> -chardev socket,id=char0,path=/tmp/vhostqemu1 \
> -device vhost-user-fs-pci,chardev=char0,tag=myfs_11,bus=pci.11 \
> -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm1,share=on \
> -numa node,memdev=mem \
> 
> 2）shutdown this vm and start it again with step1, but it failed.
> (qemu) qemu-kvm: -device
> ide-drive,bus=ide.0,unit=0,drive=drive_system_disk,id=ide0-0-0: warning:
> 'ide-drive' is deprecated, please use 'ide-hd' or 'ide-cd' instead
> qemu-kvm: -device
> ide-drive,drive=drive-ide0-1-0,id=ide0-1-0,serial=GAGENT_TEST: warning:
> 'ide-drive' is deprecated, please use 'ide-hd' or 'ide-cd' instead
> error: kvm run failed Bad address
> EAX=00000000 EBX=00000000 ECX=00000010 EDX=000efced
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00007000
> EIP=000efced EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
> LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
> TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
> GDT=     000f6200 00000037
> IDT=     000f623e 00000000
> CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
> DR3=0000000000000000 
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=47 d0 0f 00 0f b7 d2 8d 44 24 02 e8 46 c4 ff ff 83 c4 28 c3 <53> 83 ec
> 20 e8 38 c8 ff ff bb 00 00 00 40 8d 44 24 0c 50 8d 44 24 0c 50 8d 4c 24 0c
> 8d 54
> 
> 3)change mem-path to another dir, vm still couldn't boot up.
> -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm2,share=on \


pkg:
qemu-kvm-5.1.0-7.module+el8.3.0+8099+dba2fe3e.x86_64
kernel-4.18.0-236.el8.x86_64
seabios-bin-1.14.0-1.module+el8.3.0+7638+07cf13d2.noarch
Comment 4 Dr. David Alan Gilbert 2020-09-28 14:32:06 UTC 
How big is your host memory and the /dev/shm's ? I think you're just running out of RAM.
Comment 5 xiagao 2020-09-30 09:13:13 UTC 
(In reply to Dr. David Alan Gilbert from comment #4)
> How big is your host memory and the /dev/shm's ? I think you're just running
> out of RAM.

[root@dell-per730-47 dev]# cat /proc/meminfo 
MemTotal:       32623152 kB

[root@dell-per730-47 dev]# ll -lh shm1
-rw-r--r-- 1 root root 4.0G Sep 27 03:27 shm1
Comment 7 Dr. David Alan Gilbert 2020-09-30 09:41:39 UTC 
(In reply to xiagao from comment #5)
> (In reply to Dr. David Alan Gilbert from comment #4)
> > How big is your host memory and the /dev/shm's ? I think you're just running
> > out of RAM.
> 
> [root@dell-per730-47 dev]# cat /proc/meminfo 
> MemTotal:       32623152 kB
> 
> [root@dell-per730-47 dev]# ll -lh shm1
> -rw-r--r-- 1 root root 4.0G Sep 27 03:27 shm1

OK, I think that's legal, but a bit odd.
 /dev/shm1 is a file created in the /dev tmpfs filesytsem, rather than in the /dev/shm filesystem.

Still, if your host is 32G, /dev will be created as a 16G tmpfs I think, so should allow 4 4G VMs in it OK.
Comment 8 Hanna Czenczek 2020-09-30 14:28:01 UTC 
Hi,

I ran a script that I’ll attach in a minute (with the binary paths adjusted accordingly, where arch.iso is an Arch Linux ISO, and archiso.img is the initrd from that ISO) ten times and I couldn’t reproduce the problem.

(I mounted a tmpfs (8G size) to /dev/shm2.)

My host is (Arch) Linux 5.8.12, the guest kernel is 4.18.0-236.el8, the qemu version is 5.1.0-7.module+el8.3.0. Both guest kernel and qemu were compiled by me, though, and I didn’t use the RHEL configuration for the guest kernel.

The issue in comment 2 looks different, I can’t imagine virtio-fs to be the cause of KVM errors.

Max
Comment 9 Hanna Czenczek 2020-09-30 14:29:54 UTC 
Created attachment 1717901 [details]
Launch two VMs simultaneously, and mount the same host directory on both through virtio-fs
Comment 10 Hanna Czenczek 2021-03-22 13:44:39 UTC 
Hi,

Can you still reproduce this issue?  If so, can you perhaps provide a script that invokes qemu and virtiofsd in such a way to reproduce it?

Max
Comment 11 xiagao 2021-03-23 02:04:55 UTC 
(In reply to Max Reitz from comment #10)
> Hi,
> 
> Can you still reproduce this issue?  If so, can you perhaps provide a script
> that invokes qemu and virtiofsd in such a way to reproduce it?
> 
> Max

OK,I'm trying to reproduce, once it's reproduced I will let you know.
Comment 14 Hanna Czenczek 2021-03-29 10:37:29 UTC 
Hi,

I tried to reproduce again with qemu-kvm-5.2.0-8.module+el8.4.0 and Linux 4.18.0-298.el8 in the guest, but I still wasn’t “lucky”.

What’s most interesting from the automation log is that it is always exactly every second run that fails, i.e. it’s always one passing and then one failing, and again and again.  That seems suspicious to me and rather seems to indicate there might be some interaction going on between each consecutive run (i.e. passing runs leave behind something that makes the next run fail, but the failing run then cleans it up).

In comment 0, you said the reproducibility would be 4/10, though, i.e. not exactly every second run.  And you said you can manually reproduce it, too.  So was the automation log just a very lucky coincidence?

(Does the first run always pass?)

Because I can’t reproduce it, I’m afraid I’ll have to ask: Can you perhaps give me a stacktrace for the hanging virtiofsd and qemu instances?

Max
Comment 21 John Ferlan 2021-09-09 11:12:20 UTC 
Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.
Comment 22 xiagao 2022-01-19 08:35:34 UTC 
I talked with QE who is responsible for memory feature.She said Generally we don't use a general file as a memory backend file, we often use hugepage or tmpfs to be the backend file.
So I think this is not a bz.

mount hugtlbfs in host and echo memory_size of your test needed
#mount -t hugetlbfs none /mnt/hugetlbfs
# echo $memory_size > /proc/sys/vm/nr_hugepage
-m 4G /
-object memory-backend-file,host-nodes=0,policy=bind,id=mem-0,size=2048M,prealloc=yes,mem-path=/mnt/hugetlbfs -numa node,memdev=mem-0 \

I will close it,and feel free to reopen it if you have any other opinion.

Thanks.