Bug 2177628

> Description of problem:
> I run some systems in beaker for internal use.
> After upgrade rhel9 machines I ended up with unusable systems from beaker
> point of view.
> After some investigation, I figured out that console log is empty.
> But, issue from my side is, that my virtual machines were migrated from 
> rhel7 (libvirtd xml files) so deprecated pc-i440fx-rhel7.6.0 were used.

Hi Robin,
Thanks for raising this issue !
I've noticed that your migration test is between rhel7 and rhel9, right ? Actually it's not supported scenario, which has been confirmed by developers, I'd like to provide historical discussions FYI.Here is a part of email, if you want to know more I can involve you in the email thread later.
---------------------------------------------------------------------------------------------------------------------------------
So the LPs change from i440fx to Q35 on RHEL8 (RHV) or RHEL9 (OSP),
-Our current test plan has included the change and we focus on it 
while RHEL7 to RHEL9 (RHEL KVM) is not possible without changing the machine type.
IMHO this is enough to not let any customer assume there is an ABI guarantee when moving from 7 to 9, both on RHEL and LPs. 
-Stable Guest ABI test between RHEL7 and RHEL9 this matrix is not included in our current test plan. 
---------------------------------------------------------------------------------------------------------------------------------
If you have any concerns please let me know. Thanks
And also involved David and Germano.

Min

Hi.

To be clear:
1) I just grabbed libvirtd xml files from rhel7 (virsh dumpxml)
2) reinstalled machine to rhel9 (fresh install)
3) I defined machines from libvirtd xml files ... it worked fine

Then I upgraded rhel9 (dnf upgrade) and it stopped to work.

Thank you for clarification. So migrating from pc-i440fx-rhel7.6.0 to Q35 is way how to do it.

I looked like regression, because in initial state, everything worked flawlessly and after upgrade
I have had tons of machines in beaker marked as broken.

Thank you for your help and if it's not real bug, then feel free to close it or I going to close it today.


(In reply to Min Deng from comment #1)
> > Description of problem:
> > I run some systems in beaker for internal use.
> > After upgrade rhel9 machines I ended up with unusable systems from beaker
> > point of view.
> > After some investigation, I figured out that console log is empty.
> > But, issue from my side is, that my virtual machines were migrated from 
> > rhel7 (libvirtd xml files) so deprecated pc-i440fx-rhel7.6.0 were used.
> 
> Hi Robin,
> Thanks for raising this issue !
> I've noticed that your migration test is between rhel7 and rhel9, right ?
> Actually it's not supported scenario, which has been confirmed by
> developers, I'd like to provide historical discussions FYI.Here is a part of
> email, if you want to know more I can involve you in the email thread later.
> -----------------------------------------------------------------------------
> ----------------------------------------------------
> So the LPs change from i440fx to Q35 on RHEL8 (RHV) or RHEL9 (OSP),
> -Our current test plan has included the change and we focus on it 
> while RHEL7 to RHEL9 (RHEL KVM) is not possible without changing the machine
> type.
> IMHO this is enough to not let any customer assume there is an ABI guarantee
> when moving from 7 to 9, both on RHEL and LPs. 
> -Stable Guest ABI test between RHEL7 and RHEL9 this matrix is not included
> in our current test plan. 
> -----------------------------------------------------------------------------
> ----------------------------------------------------
> If you have any concerns please let me know. Thanks
> And also involved David and Germano.
> 
> Min

Hi Robin,

(In reply to Robin Hack from comment #2)
> Hi.
> 
> To be clear:
> 1) I just grabbed libvirtd xml files from rhel7 (virsh dumpxml)
> 2) reinstalled machine to rhel9 (fresh install)
> 3) I defined machines from libvirtd xml files ... it worked fine

Hold on. There is no way to go from RHEL7 to RHEL9 without doing changes in the XML.

Were you really on RHEL7 (qemu-kvm-1.5.3) or were you on RHEL7's qemu-kvm-rhev branch that is only to be used by RHV and OSP?
You mention pc-i440fx-rhel7.6.0, thats only supported on qemu-kvm-rhev. So was this system sort of a Frankenstein RHEL7 KVM host using RHV packages?
 
> Then I upgraded rhel9 (dnf upgrade) and it stopped to work.
This sounds like a real problem. If it was working on 9 and then dnf update to latest 9 broke something, then I think we need to investigate it.
It could perhaps not be related to the origin (RHEL7) of the VM or any deprecated settings, and we may have a bug on RHEL9.

Do you still have any of these systems in broken state so I can take a look?

Thanks

(In reply to Germano Veit Michel from comment #3)
> Hi Robin,
> 
> (In reply to Robin Hack from comment #2)
> > Hi.
> > 
> > To be clear:
> > 1) I just grabbed libvirtd xml files from rhel7 (virsh dumpxml)
> > 2) reinstalled machine to rhel9 (fresh install)
> > 3) I defined machines from libvirtd xml files ... it worked fine
> 
> Hold on. There is no way to go from RHEL7 to RHEL9 without doing changes in
> the XML.
> 
> Were you really on RHEL7 (qemu-kvm-1.5.3) or were you on RHEL7's
> qemu-kvm-rhev branch that is only to be used by RHV and OSP?
> You mention pc-i440fx-rhel7.6.0, thats only supported on qemu-kvm-rhev. So
> was this system sort of a Frankenstein RHEL7 KVM host using RHV packages?
>  
> > Then I upgraded rhel9 (dnf upgrade) and it stopped to work.
> This sounds like a real problem. If it was working on 9 and then dnf update
> to latest 9 broke something, then I think we need to investigate it.
> It could perhaps not be related to the origin (RHEL7) of the VM or any
> deprecated settings, and we may have a bug on RHEL9.

It looks like there's a similar bug for this issue, please refer to 2177636 reported by Robin.
Thanks
Min

> 
> Do you still have any of these systems in broken state so I can take a look?
> 
> Thanks

According to comment2, it's related about serial feature but not stable guest abi.
Hi NaNa,
It's another similar bug 2177636, feel free to update this bug if you need to, thank you !
Thanks
Min

(In reply to Germano Veit Michel from comment #3)
> Hi Robin,
> 
> (In reply to Robin Hack from comment #2)
> > Hi.
> > 
> > To be clear:
> > 1) I just grabbed libvirtd xml files from rhel7 (virsh dumpxml)
> > 2) reinstalled machine to rhel9 (fresh install)
> > 3) I defined machines from libvirtd xml files ... it worked fine
> 
> Hold on. There is no way to go from RHEL7 to RHEL9 without doing changes in
> the XML.
> 
> Were you really on RHEL7 (qemu-kvm-1.5.3) or were you on RHEL7's
> qemu-kvm-rhev branch that is only to be used by RHV and OSP?
> You mention pc-i440fx-rhel7.6.0, thats only supported on qemu-kvm-rhev. So
> was this system sort of a Frankenstein RHEL7 KVM host using RHV packages?
>  
Hi. I still have rhel7 machine in production.
And I have rhel9 machine with updated xml and xml are saved on hdd on this machine.

And it looks like, we have Frankenstein here because I see, on rhel7 machine:

qemu-kvm-rhev-2.9.0-16.el7_4.3.x86_64
qemu-kvm-common-rhev-2.9.0-16.el7_4.3.x86_64

and it works fine.

> > Then I upgraded rhel9 (dnf upgrade) and it stopped to work.
> This sounds like a real problem. If it was working on 9 and then dnf update
> to latest 9 broke something, then I think we need to investigate it.
> It could perhaps not be related to the origin (RHEL7) of the VM or any
> deprecated settings, and we may have a bug on RHEL9.
Yes. It really happened after upgrade. But it seems that path of less resistance is way to ... broken system :).
> 
> Do you still have any of these systems in broken state so I can take a look?
I can give you access to one hypervisor, where you can play with 1 virtual machine which I can reserve for you.
But it going to need to:
1) save machine configuration -  I can handle it
2) redefine this machine - I can handle it
3) and try to start and listen to console - if you want to investigate, then it's your task
> 
> Thanks

Thank you very much for your time.

a) Do you have XML for a before-after for the same VM; each bz seems to have one XML o nit; a matched pair would be useful
b) If you do a 'dmesg' in the two cases in the guest, is there any difference in:
    dmesg|grep ttyS
c) If on the failing config, in the guest you try:
     echo hello0 > /dev/ttyS0
     echo hello1 > /dev/ttyS1
   etc  do you get the output?

Dave

Hi. Ad b)

I need to remove machine from beaker and then recreate it from old config.
Work in progress (I need to wait for beaker to finish task/tasks).

After dicussion with Robin on gchat, this should be one configuration issue. See https://bugzilla.redhat.com/show_bug.cgi?id=2177636#c7.
And we need to set the console port value to '0' since system will send the bios log to the ttyS0 default.


Closed this bug as NOTBUG, thanks.