Bug 252301

Summary: kernel-xen 2.6.21-2925.15.fc8 regressions
Product: [Fedora] Fedora Reporter: Adam Tkac <atkac>
Component: kernel-xenAssignee: Eduardo Habkost <ehabkost>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: low Docs Contact:
Priority: low    
Version: rawhideCC: jhrozek, ovasik, vikigoyal, xen-maint
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: 2.6.20-2934.fc7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2007-09-25 08:28:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
common ops (from dmesg immediately after boot)
none
errors during paravirt start
none
dmesg from 2925.14.fc8
none
2925.14.fc8 xm dmesg
none
2931.fc8 dmesg
none
2931.fc8 xm dmesg
none
fresh ops from up2date paravirt fc7 guest none

Description Adam Tkac 2007-08-15 07:29:39 UTC
Description of problem:
I've updated to 2.6.21-2925.15.fc8 from 2.6.21-2925.14.fc8 and some things get
broken.
- Xserver restarts sometimes
- installation of RHEL4 fullvirt machine results in kernel faults (rhel4 guest
kernel fail)

Version-Release number of selected component (if applicable):
2.6.21-2925.15.fc8

How reproducible:
always

Comment 1 Adam Tkac 2007-08-15 07:38:40 UTC
Created attachment 161334 [details]
common ops (from dmesg immediately after boot)

Comment 2 Adam Tkac 2007-08-15 07:42:16 UTC
I will provide more debuginfo after next crashes

Comment 3 Adam Tkac 2007-08-15 10:45:15 UTC
Hm, kernel now completely hangs. After reboot nothing interesting in log. Any
hints how debug this?

Comment 4 Adam Tkac 2007-08-15 10:45:47 UTC
(In reply to comment #3)
> Hm, kernel now completely hangs. After reboot nothing interesting in log. Any
> hints how debug this?

Of course, during creating paravirt machine

Comment 5 Eduardo Habkost 2007-08-20 13:45:49 UTC
(In reply to comment #1)
> Created an attachment (id=161334) [edit]
> common ops (from dmesg immediately after boot)
> 


Do you have the remaining of the dmesg contents? Especially the ACPI and 
IRQ-related messages.

Is the TG3 network interface working, even with the warnings?


If you have the full contents of dmesg and 'xm dmesg' of both 
2.6.21-2925.14.fc8 and 2.6.21-2925.15.fc8, they will be useful for checking 
what changed from one version to another.

Comment 6 Eduardo Habkost 2007-08-20 20:39:59 UTC
A new kernel-xen was submitted including a fix on the kernel IRQ handling. 
This may help to solve some of the issues, especially the TG3 
synchronize_irq() warning.

Could you test kernel-xen-2.6-2.6.21-2931.fc8 (that is building right now and 
should go to Rawhide soon) and report if the problems remain?

Comment 7 Adam Tkac 2007-08-23 19:57:58 UTC
Created attachment 172373 [details]
errors during paravirt start

I don't have direct access to that machine now so I will test next kernel and
attach dmesg outputs during monday

Comment 8 Eduardo Habkost 2007-08-23 20:11:47 UTC
(In reply to comment #7)
> Created an attachment (id=172373) [edit]
> errors during paravirt start

Are those errors from the guest or from the dom0 kernel?

Comment 9 Eduardo Habkost 2007-08-23 20:53:39 UTC
(In reply to comment #7)
> Created an attachment (id=172373) [edit]
> errors during paravirt start

Do you have the output of 'xm dmesg' of when this error was generated? The 
hypervisor seem to have returned an error to the kernel, and probably 
additional info is on xen dmesg.

Comment 10 Adam Tkac 2007-08-30 10:38:57 UTC
I did some play with kernels 2925.14.fc8 and 2931.fc8. Kernel always hangs when
multiple domains are under load (for example multiple domains are started, yum
update on multiple domains etc). About comment #7 : If I could remember
correctly I started machine remotely (ssh + virsh) and remote machine (means
dom0, not domU) has to be rebooted so I guess problem could be with dom0.

Comment 11 Adam Tkac 2007-08-30 10:39:44 UTC
Created attachment 180541 [details]
dmesg from 2925.14.fc8

Comment 12 Adam Tkac 2007-08-30 10:40:27 UTC
Created attachment 180561 [details]
2925.14.fc8 xm dmesg

Comment 13 Adam Tkac 2007-08-30 10:40:46 UTC
Created attachment 180581 [details]
2931.fc8 dmesg

Comment 14 Adam Tkac 2007-08-30 10:41:02 UTC
Created attachment 180601 [details]
2931.fc8 xm dmesg

Comment 15 Adam Tkac 2007-09-06 09:07:16 UTC
I've removed Regression keyword because it seems problems were also in older kernel

Comment 16 Adam Tkac 2007-09-06 09:08:20 UTC
Created attachment 188501 [details]
fresh ops from up2date paravirt fc7 guest

Comment 17 Eduardo Habkost 2007-09-13 18:12:19 UTC
(In reply to comment #7)
> Created an attachment (id=172373) [edit]
> errors during paravirt start
>

The log contains:

> kernel: EIP is at xen_l1_entry_update+0x95/0xa1

I just got a xen_l2_entry_update() Oops on my test machine,
and it is related CONFIG_HIGHPTE, that is not really reliable under Xen, but 
it was enabled on Fedora. I have disabled CONFIG_HIGHPTE on Rawhide kernel-xen 
(kernel-xen-2.6-2.6.21-2937.fc8) and it will be disabled on the next F7 
kernel-xen update (kernel-xen-2.6-2.6.20-2934.fc7 or higher).

The oops from comment #16 is also a symptom of the same bug related HIGHPTE.


Could you check if kernel-xen-2.6-2.6.21-2937.fc8 is more stable, for you?

If the stability problems are solved, then we have only the tg3 
synchronize_irq() problems.

Comment 18 Adam Tkac 2007-09-17 13:15:14 UTC
I tried 2.6.21-2940.fc8 and it looks fine. Kernel has no problems under heavy
load. Only tg3 issue will be seen in dmesg output but I have no problems related
to this

Comment 19 Eduardo Habkost 2007-09-17 16:36:53 UTC
Thanks.
I will create a separated bug for the tg3 synchronize_irq() warning.

Comment 20 Eduardo Habkost 2007-09-17 16:46:15 UTC
Bug #293451 created for the tg3 synchronize_irq() warning.

I am closing this bug, as the stability problems caused by HIGHPTE are gone.

Comment 21 Fedora Update System 2007-09-18 03:13:58 UTC
kernel-xen-2.6-2.6.20-2934.fc7 has been pushed to the Fedora 7 testing repository.  If problems still persist, please make note of it in this bug report.

Comment 22 Fedora Update System 2007-09-25 08:28:47 UTC
kernel-xen-2.6-2.6.20-2934.fc7 has been pushed to the Fedora 7 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 23 Eduardo Habkost 2007-10-17 12:55:04 UTC
*** Bug 335961 has been marked as a duplicate of this bug. ***

Comment 24 Eduardo Habkost 2007-10-22 14:33:34 UTC
*** Bug 254941 has been marked as a duplicate of this bug. ***