Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2233350

Summary: Failed to migrate rhel79 guest with pc machine between rhel 8.9 and rhel9.3
Product: Red Hat Enterprise Linux 9 Reporter: Min Deng <mdeng>
Component: qemu-kvmAssignee: Peter Xu <peterx>
qemu-kvm sub component: Machine Types QA Contact: Min Deng <mdeng>
Status: CLOSED MIGRATED Docs Contact:
Severity: high    
Priority: high CC: coli, fjin, germano, gveitmic, jinzhao, juzhang, lijin, mdeng, nilal, peterx, quintela, virt-maint, zhguo
Version: 9.3Keywords: CustomerScenariosInitiative, MigratedToJIRA, Regression, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-22 18:10:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Min Deng 2023-08-22 03:30:33 UTC
Description of problem:
Failed to migrate rhel79 guest with pc machine between rhel 8.9 to rhel9.3 

Version-Release number of selected component (if applicable):
RHEL 8.9.0:
kernel-4.18.0-510.el8.x86_64
qemu-kvm-6.2.0-38.module+el8.9.0+19636+489b90af.x86_64
seabios-bin-1.16.0-4.module+el8.9.0+19570+14a90618.noarch
edk2-ovmf-20220126gitbb1bba3d77-6.el8.noarch
RHEL 9.3
kernel-5.14.0-355.el9.x86_64
qemu-kvm-8.0.0-12.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch

How reproducible:
3/3
Steps to Reproduce:
1.boot up a guest 
/usr/libexec/qemu-kvm -name "mouse-vm" -sandbox on \
-name guest=rhel79,debug-threads=on \
-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on \
-accel kvm \
-cpu Broadwell-noTSX,ibpb=on,md-clear=on,spec-ctrl=on,ssbd=on,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on \
-m size=4194304k \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296}' \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=4,threads=1 \
-uuid 509b0c90-72a1-4ce3-acbc-7a72211ab221 \
-no-user-config \
-nodefaults \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot strict=on \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x4"}' \
-device '{"driver":"virtio-scsi-pci","id":"scsi0","bus":"pci.0","addr":"0x5"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x6"}' \
-blockdev '{"driver":"file","filename":"rhel79-64-virtio-scsi.qcow2.seabios","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":0,"device_id":"drive-scsi0-0-0-0","drive":"libvirt-1-format","id":"scsi0-0-0-0","bootindex":1,"write-cache":"on"}' \
-device '{"driver":"virtio-net-pci","id":"net0","mac":"52:56:00:00:00:11","bus":"pci.0","addr":"0x3"}' \
-chardev file,id=charserial0,path=/tmp/t11,append=on \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev pty,id=charserial1 \
-device '{"driver":"isa-serial","chardev":"charserial1","id":"serial1","index":1}' \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,audiodev=audio1 \
-device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pci.0","addr":"0x2"}' \
-incoming defer \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.0","addr":"0x7"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.0","addr":"0x8"}' \
-monitor stdio
2.migrate it with pc-i440fx-rhel7.6.0 for RHEL9.3 

Actual results:
QEMU 8.0.0 monitor - type 'help' for more information
(qemu) migrate_incoming tcp:[::]:4000
(qemu) qemu-kvm: Missing section footer for 0000:00:01.3/piix4_pm
qemu-kvm: load of migration failed: Invalid argument
Expected results:
Migration successfully

Additional info:

Comment 2 Peter Xu 2023-08-23 15:30:55 UTC
A few questions to Min:

1. It seems you're using rhel76-pc, but you mentioned rhel76 in the subject and comment 0.  I assume you really meant rhel76 guest, right?

2. Is it just start to fail on 9.3?  What happens if you change src/dst?  E.g., 8.9->9.2/9.1/9.0?

3. Could you help me collect the vmstate dump on both sides?  To do that, you need to do:

  (a) On src host, append your qemu cmdline with "-dump-vmstate vmstate.src" and run that command.  
      Instead of starting the VM it should generate a file called "vmstate.src" under the same directory.
   
  (b) On dst host, append your qemu cmdline with "-dump-vmstate vmstate.dst" and run similarly, generating vmstate.dst.

  Then share the two files.

Thanks,
Peter

Comment 3 Min Deng 2023-08-29 08:47:51 UTC
Hi Peter,
Sorry for delay, I was in sick leave previous days.

> 1. It seems you're using rhel76-pc, but you mentioned rhel76 in the subject
> and comment 0.  I assume you really meant rhel76 guest, right?

It's rhel79 guest

> 
> 2. Is it just start to fail on 9.3?  What happens if you change src/dst? 
> E.g., 8.9->9.2/9.1/9.0?
> 
> 3. Could you help me collect the vmstate dump on both sides?  To do that,
> you need to do:
> 
>   (a) On src host, append your qemu cmdline with "-dump-vmstate vmstate.src"
> and run that command.  
>       Instead of starting the VM it should generate a file called
> "vmstate.src" under the same directory.
>    
>   (b) On dst host, append your qemu cmdline with "-dump-vmstate vmstate.dst"
> and run similarly, generating vmstate.dst.
> 
>   Then share the two files.

Yes, will do it and provide the result. Set needinfo to myself

Comment 5 Min Deng 2023-09-04 03:33:39 UTC
Tried the bug between rhel 8.8 and rhel 9.3
QE didn't reproduce the issue.
RHEL 8.8
kernel-4.18.0-477.26.1.el8_8.x86_64
qemu-kvm-6.2.0-33.module+el8.8.0+19768+98f68f21.x86_64
seabios-1.16.0-4.module+el8.8.0+19627+2d14cb21.x86_64
seabios-bin-1.16.0-4.module+el8.8.0+19627+2d14cb21.noarch
RHEL 9.3
kernel-5.14.0-355.el9.x86_64
qemu-kvm-8.0.0-12.el9.x86_64
qemu-kvm-tests-8.0.0-12.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
seabios-1.16.1-1.el9.x86_64

Comment 6 Min Deng 2023-09-05 09:05:36 UTC
Tried the bug [the pc machine type] from rhel 8.8 to rhel 8.9,I can also reproduce the issue. I will file a new issue for rhel8 correspondingly. Thank you !
RHEL.8.8.0
4.18.0-477.26.1.el8_8.x86_64
qemu-kvm-6.2.0-33.module+el8.8.0+19768+98f68f21.x86_64
seavgabios-bin-1.16.0-4.module+el8.9.0+19570+14a90618.noarch
RHEL.8.9.0
kernel-4.18.0-510.el8.x86_64
qemu-kvm-6.2.0-38.module+el8.9.0+19636+489b90af.x86_64
seavgabios-bin-1.16.0-4.module+el8.9.0+19570+14a90618.noarch

Comment 7 Peter Xu 2023-09-06 15:43:23 UTC
Min,

Did you collect the thing I asked in comment 2?

You can also share me the hosts for a few hours and I can collect them myself.

Thanks.

Comment 8 Min Deng 2023-09-06 17:19:21 UTC
(In reply to Peter Xu from comment #7)
> Min,
> 
> Did you collect the thing I asked in comment 2?
> 
> You can also share me the hosts for a few hours and I can collect them
> myself.
> 
> Thanks.
Please refer to comment 4, thank you!

Comment 9 Peter Xu 2023-09-06 18:19:19 UTC
Oh yes, sorry I missed that.

I think the static checker reported something indeed, but I'm not yet sure this is relevant:

Section "qemu-xhci", Description "xhci-event": expected field "cve_2014_5263_a", while dest has no further fields
Section "nec-usb-xhci", Description "xhci-event": expected field "cve_2014_5263_a", while dest has no further fields
Section "pci-xhci", Description "xhci-event": expected field "cve_2014_5263_a", while dest has no further fields
Section "e1000e", Description "e1000e": expected field "redhat_7_3_intr_state", got "core.rxbuf_min_shift"; skipping rest

At least cve_2014_5263_a is only set for rhel 7.0 guests not 7.9.

I'll have a closer look.

Comment 10 Peter Xu 2023-09-06 20:07:56 UTC
OK I think we miss this commit for all rhel8 branches:

commit a83c2844903c45aa7d32cdd17305f23ce2c56ab9
Author: Dr. David Alan Gilbert <dgilbert>
Date:   Wed Apr 6 14:58:12 2022 -0400

    acpi: fix acpi_index migration

even 9.0.0 misses it.   But 9.3.0 (the version this bug originates) should have it.

Min, I see that your testing results seem to be contradictory between each other.. so I want to double check with you:

  - In comment 0, you said 8.9->9.3 fails, so this bug is opened
  - In comment 5, you said 8.9->9.3 is fine.

Which one is correct?  If comment 5 is correct, should we close this bug (while keeping https://issues.redhat.com/browse/RHEL-2186 to track the real bug)?

Comment 21 Min Deng 2023-09-21 09:10:57 UTC
Hi All,
Will we fix this issue for rhel 9.3 or next release, thank you !
Best Regards,
Min

Comment 22 Nitesh Narayan Lal 2023-09-21 15:05:10 UTC
(In reply to Min Deng from comment #21)
> Hi All,
> Will we fix this issue for rhel 9.3 or next release, thank you !
> Best Regards,
> Min

Hi Min, I was under the impression that on rhel9, the issue is only present in 9.0. Is that not the case?

Comment 23 RHEL Program Management 2023-09-22 18:09:04 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 24 RHEL Program Management 2023-09-22 18:10:35 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.