Bug 817236

Summary: Regression since 2.6.32-266.el6 AMD host writes 150+ GB dmesg logs
Product: Red Hat Enterprise Linux 6 Reporter: Xiaoqing Wei <xwei>
Component: kernelAssignee: Marcelo Tosatti <mtosatti>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: atodorov, juzhang, knoel, michen, qzhang, shuang, sluo, tburke, yunzheng
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-2.6.32-270.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-20 14:00:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xiaoqing Wei 2012-04-28 07:49:21 UTC
Description of problem:
Regression since 2.6.32-266.el6  kernel writes 150+ GB dmesg logs
Version-Release number of selected component (if applicable):

kernel-2.6.32-268.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. install a host with kernel >= kernel-2.6.32-266.el6.x86_64, here i am using 268
2. boot a kvm guest
3. the below dmesg repeat again and again, till disk ran out of space


  ------------[ cut here ]------------
WARNING: at arch/x86/kvm/svm.c:729 svm_adjust_tsc_offset+0xac/0xc0 [kvm_amd]() (Tainted: G        W  ---------------   )
Hardware name: OptiPlex 740 Enhanced
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand powernow_k8 freq_table mperf bridge stp llc ipv6 vhost_net macvtap macvlan tun kvm_amd kvm ppdev parport_pc parport microcode dcdbas serio_raw edac_core edac_mce_amd k8temp snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc sg i2c_nforce2 tg3 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic sata_nv nouveau ttm drm_kms_helper drm i2c_core mxm_wmi video output wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 7729, comm: qemu-kvm Tainted: G        W  ---------------    2.6.32-266.el6.x86_64 #1
Call Trace:
 [<ffffffff8106b6b7>] ? warn_slowpath_common+0x87/0xc0
 [<ffffffff8106b70a>] ? warn_slowpath_null+0x1a/0x20
 [<ffffffffa03cb21c>] ? svm_adjust_tsc_offset+0xac/0xc0 [kvm_amd]
 [<ffffffffa03736cb>] ? kvm_arch_vcpu_load+0xdb/0x140 [kvm]
 [<ffffffffa0369db5>] ? vcpu_load+0x55/0x80 [kvm]
 [<ffffffffa0373985>] ? kvm_arch_destroy_vm+0x65/0x170 [kvm]
 [<ffffffffa03687f4>] ? kvm_put_kvm+0xe4/0x110 [kvm]
 [<ffffffffa03698c8>] ? kvm_vcpu_release+0x18/0x20 [kvm]
 [<ffffffff8117d065>] ? __fput+0xf5/0x210
 [<ffffffff8117d1a5>] ? fput+0x25/0x30
 [<ffffffff81178bcd>] ? filp_close+0x5d/0x90
 [<ffffffff8106e32f>] ? put_files_struct+0x7f/0xf0
 [<ffffffff8106e3f3>] ? exit_files+0x53/0x70
 [<ffffffff8107046d>] ? do_exit+0x18d/0x870
 [<ffffffff8108287d>] ? signal_wake_up+0x2d/0x40
 [<ffffffff81070ba8>] ? do_group_exit+0x58/0xd0
 [<ffffffff81070c37>] ? sys_exit_group+0x17/0x20
 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
---[ end trace a9a4cdc9830925b7 ]---

Actual results:
/var/log will out of space.
#du -hs /var/log/messages
149G	/var/log/messages

Expected results:
both host and guest works well, not suspicious and annoying log.

Additional info:
till now, only happen on
processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 107
model name	: AMD Athlon(tm) Dual Core Processor 5400B
stepping	: 2
cpu MHz		: 1000.000
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2

Comment 3 Xiaoqing Wei 2012-04-28 08:05:33 UTC
this probably introduced by the fix of 
Bug 807215 - after host S4 the guest can not work normally

Comment 9 Jarod Wilson 2012-05-09 16:33:00 UTC
Patch(es) available on kernel-2.6.32-270.el6

Comment 12 Marcelo Tosatti 2012-05-14 16:11:54 UTC
*** Bug 821365 has been marked as a duplicate of this bug. ***

Comment 13 Qunfang Zhang 2012-05-15 04:56:57 UTC
Reproduced on kernel-2.6.32-268.el6 and verified pass on kernel-2.6.32-270.el6.

Steps:
1. Boot a guest on host.
2. On host: # du -hs /var/log/messages 
3. Repeat step 2.

on kernel-2.6.32-268.el6, after step 3:
The file size on step 2 keep increasing and there's lots of Call Trace in the
dmesg of guest.

on kernel-2.6.32-270.el6, after step 3:
[root@intel-q9400-8-1 ~]# du -hs /var/log/messages 
116K /var/log/messages

And this bug is reproduced only on an AMD host, and verified passed on both
Intel and AMD host with RHEL6.3 and winxp guests.

So, this bug is fixed, setting to VERIFIED.

Comment 15 errata-xmlrpc 2012-06-20 14:00:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0862.html