Bug 2212734

Summary: [RHEL9.3][virt-install][dasd][s390x] there will be an xfs I/O error when install guest into a used dasd with same vm name
Product: Red Hat Enterprise Linux 9 Reporter: bfu <bfu>
Component: virt-managerAssignee: Thomas Huth <thuth>
virt-manager sub component: Common QA Contact: Hongzhou Liu <hongzliu>
Status: CLOSED WORKSFORME Docs Contact:
Severity: high    
Priority: high CC: bfiuczyn, clegoate, cohuck, crobinso, dhorak, hannsj_uhl, hongzliu, jinzhao, juzhang, juzhou, knoel, lcong, lijin, pbonzini, pkrempa, ribarry, smitterl, stefanha, thuth, tyan, tzheng, vgoyal, virt-maint, virt-qe-z, yiwei
Version: 9.3   
Target Milestone: rc   
Target Release: ---   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-20 09:16:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description bfu 2023-06-06 07:54:25 UTC
Description of problem:
there will be an xfs I/O error when install guest into a used dasd with same vm name.


Version-Release number of selected component (if applicable):
libvirt version: libvirt-9.3.0-2.el9.s390x
qemu version: qemu-kvm-8.0.0-2.el9.s390x
kernel version: kernel-5.14.0-316.el9.s390x

How reproducible:
100%

Steps to Reproduce:
1. install a vm into a dasd as virtio-blk
virt-install --memory 4096 --name dasd --nographics --vcpus 4 --disk none --disk /dev/dasda --disk size=20 --cdrom /home/isos/RHEL-9.3.0-20230521.45-s390x-dvd1.iso --os-variant rhel9-unknown

2. shutdown the vm and install trigger the command aganin
virt-install --memory 4096 --name dasd --nographics --vcpus 4 --disk none --disk /dev/dasda --disk size=20 --cdrom /home/isos/RHEL-9.3.0-20230521.45-s390x-dvd1.iso --os-variant rhel9-unknown


Actual results:
[    1.844574] XFS (dm-0): Metadata corruption detected at xfs_agf_verify+0x3e/0x220 [xfs], xfs_agf block 0x660008
[    1.844696] XFS (dm-0): Unmount and run xfs_repair
[    1.844697] XFS (dm-0): First 128 bytes of corrupted metadata buffer:
[    1.844699] 00000000: 58 41 47 46 00 00 00 01 00 00 00 01 00 0c c0 00  XAGF............
[    1.844700] 00000010: 00 00 00 04 00 00 00 05 00 00 00 00 00 00 00 01  ................
[    1.844701] 00000020: 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 04  ................
[    1.844702] 00000030: 00 00 00 04 00 0b b3 00 00 0a a6 78 00 00 00 00  ...........x....
[    1.844702] 00000040: de fa 78 49 99 01 4f 1e af 69 39 96 ee c6 3c a1  ..xI..O..i9...<.
[    1.844703] 00000050: 00 00 00 00 00 00 00 01 00 00 00 08 00 00 00 01  ................
[    1.844703] 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    1.844704] 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[    1.844706] XFS (dm-0): metadata I/O error in "xfs_read_agf+0xa4/0x120 [xfs]" at daddr 0x660008 len 8 error 117
[    1.845428] XFS (dm-0): Error -117 reserving per-AG metadata reserve pool.
[    1.845433] XFS (dm-0): Corruption of in-memory data (0x8) detected at xfs_fs_reserve_ag_blocks+0xfe/0x110 [xfs] (fs/xfs/xfs_fsops.c:583).  Shutting down filesystem.
[    1.845512] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[    1.845513] XFS (dm-0): Ending clean mount
mkdir: cannot stat '/sysroot': Input/output error
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

Expected results:
libvirt should tell me the disk /dev/dasda is already in use



Additional info:
If I'm using different vm name, libvirt would tell me:
[root@l42 home]# virt-install --memory 4096 --name dasd1 --nographics --vcpus 4 --disk none --disk /dev/dasda --disk size=20 --cdrom /home/isos/RHEL-9.3.0-20230521.45-s390x-dvd1.iso --os-variant rhel9-unknown
ERROR    Disk /dev/dasda is already in use by other guests ['dasd']. (Use --check path_in_use=off or --check all=off to override)

Comment 1 Peter Krempa 2023-06-20 14:57:38 UTC
What you are requesting doesn't make much sense to me. It's a perfectly legitimate configuration to attempt to install a VM again into the same disk or block device so virt-manager/install, libvirt or qemu have no reason to print any error. That part of this BZ is considered "NOTABUG".

Now for the XFS error, I'm not sure why the guest even prints that. The installer should not attempt to mount the block device at all until installation starts.

Could you please attach the output of the virt-install command above with '--print-xml' added, to see what boot device was selected for the installation.

Comment 2 bfu 2023-06-26 07:38:54 UTC
(In reply to Peter Krempa from comment #1)
> What you are requesting doesn't make much sense to me. It's a perfectly
> legitimate configuration to attempt to install a VM again into the same disk
> or block device so virt-manager/install, libvirt or qemu have no reason to
> print any error. That part of this BZ is considered "NOTABUG".
> 
> Now for the XFS error, I'm not sure why the guest even prints that. The
> installer should not attempt to mount the block device at all until
> installation starts.
> 
> Could you please attach the output of the virt-install command above with
> '--print-xml' added, to see what boot device was selected for the
> installation.

I'm not familiar with libvirt, but my colleague from libvirt QE told my that libvirt should report the dasd is in use, and that's what happened on x86let me involve him in this bz
@lcong

Comment 4 liang cong 2023-06-27 01:51:54 UTC
if virt-install a guest with same disk file and same guest name, then there would be error prompts:
# rpm -q virt-install libvirt
virt-install-4.1.0-4.el9.noarch
libvirt-9.4.0-1.el9.x86_64


# virt-install --connect qemu:///system -n vm1 --hvm --accelerate -r 2048 --vcpus=2 --os-variant rhel9.2  --disk path=/var/lib/libvirt/images/RHEL-9.3-x86_64-latest-ovmf.qcow2,bus=virtio,format=qcow2 --network network=default,model=virtio --import --noreboot --noautoconsole --serial pty --debug --memballoon model=virtio --cpu host,disable=vmx --graphics vnc --video vga --boot uefi --machine q35
...
virt-install 65543] DEBUG (cli:256)   File "/usr/bin/virt-install", line 8, in <module>
    virtinstall.runcli()
  File "/usr/share/virt-manager/virtinst/virtinstall.py", line 1234, in runcli
    sys.exit(main())
  File "/usr/share/virt-manager/virtinst/virtinstall.py", line 1221, in main
    guest, installer = build_guest_instance(conn, options)
  File "/usr/share/virt-manager/virtinst/virtinstall.py", line 672, in build_guest_instance
    cli.validate_disk(disk)
  File "/usr/share/virt-manager/virtinst/cli.py", line 378, in validate_disk
    check_inuse_conflict()
  File "/usr/share/virt-manager/virtinst/cli.py", line 366, in check_inuse_conflict
    _optional_fail(msg, "path_in_use")
  File "/usr/share/virt-manager/virtinst/cli.py", line 319, in _optional_fail
    fail(msg + (_(" (Use --check %s=off or "
  File "/usr/share/virt-manager/virtinst/cli.py", line 256, in fail
    log.debug("".join(traceback.format_stack()))

[Mon, 26 Jun 2023 21:48:56 virt-install 65543] ERROR (cli:257) Disk /var/lib/libvirt/images/RHEL-9.3-x86_64-latest-ovmf.qcow2 is already in use by other guests ['vm1']. (Use --check path_in_use=off or --check all=off to override)

Comment 5 bfu 2023-06-27 08:20:32 UTC
(In reply to Peter Krempa from comment #1)
> What you are requesting doesn't make much sense to me. It's a perfectly
> legitimate configuration to attempt to install a VM again into the same disk
> or block device so virt-manager/install, libvirt or qemu have no reason to
> print any error. That part of this BZ is considered "NOTABUG".
> 
> Now for the XFS error, I'm not sure why the guest even prints that. The
> installer should not attempt to mount the block device at all until
> installation starts.
> 
> Could you please attach the output of the virt-install command above with
> '--print-xml' added, to see what boot device was selected for the
> installation.

Could you please check comment #4?

Comment 6 Peter Krempa 2023-07-14 12:52:33 UTC
Moving to virt-manager for further analysis, as the check is solely implemented in virt-install. From libvirt's view it is okay to use the same image or device to back multiple VMs as long as they are not used at the same time. Thus also re-installing the same VM with another one with the same definition is not a problem.

Comment 7 John Ferlan 2023-07-17 20:04:34 UTC
Cole when you get a chance could you please take a look and provide some comments / feedback

Comment 8 Thomas Huth 2023-07-18 11:58:34 UTC
Hi bfu! I now tried to reproduce this problem on my LPAR, but so far, I failed. When I ran virt-install a second time, I got:

# virt-install --memory 4096 --name dasdtest --nographics --vcpus 4 --disk none --disk /dev/dasda --disk size=20 --cdrom /tmp/RHEL-9.3.0-20230715.d.41-s390x-dvd1.iso --os-variant rhel9-unknown
ERROR    Disk /dev/dasda is already in use by other guests ['dasdtest']. (Use --check path_in_use=off or --check all=off to override)

I'm using:
host kernel: 5.14.0-338
qemu-kvm: 8.0.0-7
libvirt: 9.3.0-2
virt-install: 4.1.0-4

A couple of questions:
1) Which version of virt-install have you been using?
2) You have two "--disk" parameters -- is the 20G disk in use here, too?
3) Which kind of software selection did you install in the guest? E.g. "Server"? Or "Minimal"?
4) Could you maybe retry with the latest versions from RHEL 9.3 ?

Comment 9 bfu 2023-07-20 07:17:06 UTC
(In reply to Thomas Huth from comment #8)
> Hi bfu! I now tried to reproduce this problem on my LPAR, but so far, I
> failed. When I ran virt-install a second time, I got:
> 
> # virt-install --memory 4096 --name dasdtest --nographics --vcpus 4 --disk
> none --disk /dev/dasda --disk size=20 --cdrom
> /tmp/RHEL-9.3.0-20230715.d.41-s390x-dvd1.iso --os-variant rhel9-unknown
> ERROR    Disk /dev/dasda is already in use by other guests ['dasdtest'].
> (Use --check path_in_use=off or --check all=off to override)
> 
> I'm using:
> host kernel: 5.14.0-338
> qemu-kvm: 8.0.0-7
> libvirt: 9.3.0-2
> virt-install: 4.1.0-4
> 
> A couple of questions:
> 1) Which version of virt-install have you been using?
> 2) You have two "--disk" parameters -- is the 20G disk in use here, too?
> 3) Which kind of software selection did you install in the guest? E.g.
> "Server"? Or "Minimal"?
> 4) Could you maybe retry with the latest versions from RHEL 9.3 ?

1. virt-install-4.1.0-4.el9.noarch.rpm 
2. I selected a 15.6G disk during installation
3. server with GUI(default)
4. I'm not able to reproduce it as well now...

Comment 10 bfu 2023-07-20 09:16:54 UTC
since we cannot reproduce this one, I'll close it for now, thanks for helping