Bug 507765 - XenD looses PVFB config info after PV guest reboot [NEEDINFO]
XenD looses PVFB config info after PV guest reboot
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
5.4
All Linux
low Severity medium
: rc
: 5.4
Assigned To: Michal Novotny
Virtualization Bugs
: Regression
: 509944 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-06-24 02:15 EDT by Adam Deng
Modified: 2014-02-02 17:37 EST (History)
16 users (show)

See Also:
Fixed In Version: xen-3.0.3-91.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-02 06:11:54 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
jdenemar: needinfo? (santwana.samantray)


Attachments (Terms of Use)
virt-manager.log (41.51 KB, application/octet-stream)
2009-06-24 02:18 EDT, Adam Deng
no flags Details
xend-debug.log (1.36 KB, application/octet-stream)
2009-06-24 02:18 EDT, Adam Deng
no flags Details
sosreport (1.92 MB, application/octet-stream)
2009-06-24 02:19 EDT, Adam Deng
no flags Details
Output from 'start, dumpxml, reboot, dumpxml' (2.37 KB, text/plain)
2009-06-25 10:47 EDT, Cole Robinson
no flags Details
Output from 'virsh start, xm list --long, virsh reboot, xm list --long' (2.49 KB, text/plain)
2009-06-25 10:47 EDT, Cole Robinson
no flags Details
PVFB backend removal fix (1.72 KB, patch)
2009-07-08 11:20 EDT, Michal Novotny
no flags Details | Diff
xend.log (110.50 KB, text/plain)
2009-07-09 05:10 EDT, Adam Deng
no flags Details
PVFB backend removal fix update (2.10 KB, patch)
2009-07-14 04:59 EDT, Michal Novotny
no flags Details | Diff
New fix for this BZ (3.55 KB, patch)
2009-07-20 08:45 EDT, Michal Novotny
no flags Details | Diff

  None (edit)
Description Adam Deng 2009-06-24 02:15:05 EDT
Description of problem:
   The virtual console of PV guest in virt-manager can't work when issue reboot command while the OS can boot up(can log in to it using ssh),  while virtual console of HVM guest works well, to make the virtual console work, the work arrounds are:

*Click the "Shut Down" button on virt-manager and then click "Run" button after the PV guest shutdown.
*Issue "shutdown -h now" then click "Run" button

Version-Release number of selected component (if applicable):
[root@maxxm ~]# uname -a
Linux maxxm.rx3600-7.test 2.6.18-152.el5xen #1 SMP Wed Jun 3 19:21:01 EDT 2009 ia64 ia64 ia64 GNU/Linux
[root@maxxm ~]# rpm -qa | grep virt-manager
virt-manager-0.6.1-3.el5


How reproducible:
Always


Steps to Reproduce:
1.Install RHEL5.4 alpha1 (xen) using installation number as Dom0.
2.Install a PV guest (rhel5.4a1) as domU. 
3.Reboot the domU.(domU system command or click the virt-manager reboot button)
4.After shutdown, Virt-manager console Interface always displays :"Console not configured for guest". Actually domU have been rebooted. DomU's SSH port is open.

  
Actual results:
After reboot, the virtual console of PV guest in virt-manager can't work.

Expected results:
The virtual console of PV guest in virt-manager should display whole reboot process and work well.

Additional info:
Comment 1 Adam Deng 2009-06-24 02:18:09 EDT
Created attachment 349200 [details]
virt-manager.log
Comment 2 Adam Deng 2009-06-24 02:18:52 EDT
Created attachment 349201 [details]
xend-debug.log
Comment 3 Adam Deng 2009-06-24 02:19:41 EDT
Created attachment 349202 [details]
sosreport
Comment 6 Cole Robinson 2009-06-25 10:46:22 EDT
Looks like this is either a xen or libvirt bug (reassigning to libvirt for now). Rebooting a VM using virsh makes the <display> device completely disappear from the XML. Some attachments coming.
Comment 7 Cole Robinson 2009-06-25 10:47:12 EDT
Created attachment 349406 [details]
Output from 'start, dumpxml, reboot, dumpxml'
Comment 8 Cole Robinson 2009-06-25 10:47:47 EDT
Created attachment 349407 [details]
Output from 'virsh start, xm list --long, virsh reboot, xm list --long'
Comment 9 Cole Robinson 2009-07-07 15:12:09 EDT
*** Bug 509944 has been marked as a duplicate of this bug. ***
Comment 12 Daniel Berrange 2009-07-07 16:36:11 EDT
Aaaaaaaaaaaahh, Cole's log from comment #8 is slightly mis-leading. In the second invocation of 'xm list --long' the guest has not yet rebooted - note that the domain ID is still '10' - you'd expect it to be 11.

I can reproduce this and when it happens, 'xm list --long' ceases to give back any info about the vfb device, hence libvirt can't report the VNC info.

Can't think of anything off hand that would cause this, but it is clearly a must-fix blocker
Comment 13 Michal Novotny 2009-07-08 11:20:13 EDT
Created attachment 350951 [details]
PVFB backend removal fix

I've found out what the problem was. The problem was introduced by fix for BZ #439182 about PVFB devices xenstore leak. The problem here is that it removed the PVFB devices but XenD was unable to set the devices back again after restart so I did a patch to avoid those devices removal when restarting VMs so this is the patch not to detach PVFB devices when restarting domains.
Comment 14 Adam Deng 2009-07-09 04:31:46 EDT
(In reply to comment #13)
> Created an attachment (id=350951) [details]
> PVFB backend removal fix
> 
> I've found out what the problem was. The problem was introduced by fix for BZ
> #439182 about PVFB devices xenstore leak. The problem here is that it removed
> the PVFB devices but XenD was unable to set the devices back again after
> restart so I did a patch to avoid those devices removal when restarting VMs so
> this is the patch not to detach PVFB devices when restarting domains.  


Hi, I tried to add this patch to the package and retest it, but the issue still exist. 
This is my steps:
1.Install the xen source rpm package:xen-3.0.3-88.el5.src.rpm.
2.Copy the patch into /usr/src/redhat/SOURCES/ and modify the SPEC file "xen.spec" to add the patch.
3.Build binary and source packages. Type "rpmbuild -ba xen.spec". Then rpm package "xen-3.0.3-88.ia64.rpm" and "xen-libs-3.0.3-88.ia64.rpm" are found.
4.Type commands:
rpm -e xen-3.0.3-88.el5
rpm -e --nodeps xen-libs-3.0.3-88.el5
rpm -ivh xen-libs-3.0.3-88.ia64.rpm xen-3.0.3-88.ia64.rpm
5.After the above steps, retest it, but the issue still exist.


I am not sure whether my procedure is right. Or the patch is not in effect. Please have a look. Thanks.
Comment 15 Michal Novotny 2009-07-09 04:48:31 EDT
Well, could you grab /var/log/xen/xend.log output? I have added some more logging there so I should be able to get to know whether it's been recompiled OK. And also, I didn't test it on ia64 at all but on x86_64 only.

Anyway, did you restart xend service after installing new RPMs ?

Thanks,
Michal
Comment 16 Adam Deng 2009-07-09 05:10:30 EDT
Created attachment 351029 [details]
xend.log
Comment 17 Adam Deng 2009-07-09 05:16:17 EDT
(In reply to comment #15)
Yes. After installing the new package(step 4 in Comment #14). I reboot the system, then start to test(step 5 in Comment #14).

Comment #16 has xend.log file.

Thanks for your reply.
Comment 18 Michal Novotny 2009-07-09 05:50:10 EDT
Ok, you're having my patch applied (according to "Restart in progress" contents). What's strange is that this is set to False but the line above says the domain is shutdown because of it's rebooting. Apparently I can't trust already existing entry in xenstore that should mean the domain is restarting. Hopefully I will find a working solution soon. The strange thing here is that I tried it on my Xeon workstation (x86_64 system) and it was working fine. May work in some circumstances which is certainly not good so give me some time to come up with some new solution.
Comment 20 Michal Novotny 2009-07-14 04:59:42 EDT
Created attachment 351563 [details]
PVFB backend removal fix update

This is updated version of my patch. The previous version was working in some conditions and in some it didn't so I chose a different approach and created a new variable to identify whether restart is in progress or not (currently existing variable had some problems because sometimes it was removed before the code went to part required for this patch). It's been tested in series of about 20 PV guest reboots and it was working fine.

In about half one hour - after brew finished the job - the RPMs with this patch applied will be available at http://people.redhat.com/minovotn/xen as -89mig version of xen RPMs - compiled for all available platforms so please do some testing with those RPMs...

Thanks,
Michal
Comment 21 Michal Novotny 2009-07-14 05:07:10 EDT
Well, RPMs are on URL as described at comment #20. Could you please try with those RPMs Adam ?

Thanks,
Michal
Comment 22 Adam Deng 2009-07-14 05:13:57 EDT
Yes, of course. I will test it later.
Comment 23 Michal Novotny 2009-07-14 05:31:51 EDT
Thanks Adam, comment this BZ then please.
Comment 24 Jiri Denemark 2009-07-14 07:27:47 EDT
Hello Santwana,

Could you please test the package from comment #20 and check that this patch does not reintroduce your problems with restarting Xen guests (https://bugzilla.redhat.com/show_bug.cgi?id=439182)

Thanks,

Jirka
Comment 26 Adam Deng 2009-07-15 02:42:26 EDT
Hi Michal, the patch from comment #20 has fixed this issue. It works well when PV guest reboot. Thanks Michal.
Comment 27 Michal Novotny 2009-07-20 08:45:51 EDT
Created attachment 354341 [details]
New fix for this BZ

Unfortunately it seems this fix broke some other things, most certainly BZ #486157 too. So this is the new updated version of this fix that should work fine. You can find test RPMs with this patch applied at:

http://people.redhat.com/minovotn/xen

Please provide test results after you finish your testing ...

Thanks,
Michal
Comment 28 Adam Deng 2009-07-20 22:29:54 EDT
I updated xen-3.0.3-90mig.el5.ia64.rpm and xen-libs-3.0.3-90mig.el5.ia64.rpm, rebooted the system dom0, and then tested it. This issue didn't be reproduce.
Comment 29 Michal Novotny 2009-07-21 05:15:30 EDT
Thanks for your testing Adam. This should be the package that Santwana should test as well because it contains the newest version of this patch.
Comment 30 Jiri Denemark 2009-07-22 11:59:20 EDT
Fix built into xen-3.0.3-91.el5
Comment 32 Yufang Zhang 2009-07-27 06:31:48 EDT
Hi,I can`t reproduce in xen-3.0.3-80.el5.And I test the bug in both xen-3.0.3-80.el5 and xen-3.0.3-91.el5 on a ia64 machine.In both cases,the virtual console of PV guest in virt-manager displays whole reboot process and works well.
Has this bug been fixed in xen-3.0.3-80.el5?
Comment 33 Jiri Denemark 2009-07-27 06:38:02 EDT
This was a regression since 5.3, which means that it was working in xen-3.0.3-80.el5. Then it was broken during the development and now, in xen-3.0.3-91.el5, it should work again.
Comment 36 Yewei Shao 2009-07-28 04:16:12 EDT
Verified on xen-3.0.3-91.el5
Comment 37 Chris Lalancette 2009-07-30 07:55:30 EDT
*** Bug 509099 has been marked as a duplicate of this bug. ***
Comment 39 errata-xmlrpc 2009-09-02 06:11:54 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1328.html

Note You need to log in before you can comment on or make changes to this bug.