Bug 810713 - Complete system hang triggered by GFS2 bug...
Complete system hang triggered by GFS2 bug...
Status: CLOSED UPSTREAM
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
16
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Ben Marzinski
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-08 11:04 EDT by Paulo Castro
Modified: 2012-11-27 05:01 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-27 05:01:24 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Paulo Castro 2012-04-08 11:04:46 EDT
Description of problem:

Whilst trying to synchronise data across a GFS filesystem and a USB disk using unison there comes a point where the whole system grinds to a hard lock.

Running an fsck.gfs2 on the FS doesn't seem to shed any light on anything particularly wrong.

Using netconsole I was able to have access to the kernels last screams of horror...

Version-Release number of selected component (if applicable):

gfs2-cluster-3.1.3-1.fc16.x86_64
gfs2-utils-3.1.3-1.fc16.x86_64
kernel-3.2.10-3.fc16.x86_64

How reproducible:

Every time.

Steps to Reproduce:
1. Mount the GFS fs
2. Start unison synchronisation
3. Start something else that might access the GFS fs like a watcher for git repo changes

Actual results:

System hard lock. A couple of minutes past and I witness an increase in CPU activity by relating with the speed of the fans increasing.
The system hands in such a violent way that no log ever gets written anywhere.
The only way I could get it to spew something was by using netconsole.

Expected results:

I would expect none of this to happen :p

Additional info:

Please let me know if there's more info I can supply.
Here follows the netconsole dump...

Apr  6 18:30:18 dellix [ 5358.185950] ------------[ cut here ]------------
Apr  6 18:30:18 dellix [ 5358.185967] kernel BUG at fs/gfs2/glops.c:200!
Apr  6 18:30:18 dellix [ 5358.185971] invalid opcode: 0000 [#1]
Apr  6 18:30:18 dellix SMP
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.185979] CPU 1
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.185982] Modules linked in:
Apr  6 18:30:18  binfmt_misc
Apr  6 18:30:18  gfs2
Apr  6 18:30:18  dlm
Apr  6 18:30:18  sha256_generic
Apr  6 18:30:18  dm_crypt
Apr  6 18:30:18  netconsole
Apr  6 18:30:18  nvidia(P)
Apr  6 18:30:18  ebtable_nat
Apr  6 18:30:18  ebtables
Apr  6 18:30:18  ipt_MASQUERADE
Apr  6 18:30:18  iptable_nat
Apr  6 18:30:18  nf_nat
Apr  6 18:30:18  xt_CHECKSUM
Apr  6 18:30:18  iptable_mangle
Apr  6 18:30:18  tun
Apr  6 18:30:18  bridge
Apr  6 18:30:18  stp
Apr  6 18:30:18  llc
Apr  6 18:30:18  drbd
Apr  6 18:30:18  lockd
Apr  6 18:30:18  lru_cache
Apr  6 18:30:18  ip6t_REJECT
Apr  6 18:30:18  nf_conntrack_ipv6
Apr  6 18:30:18  nf_defrag_ipv6
Apr  6 18:30:18  nf_conntrack_ipv4
Apr  6 18:30:18  ip6table_filter
Apr  6 18:30:18  nf_defrag_ipv4
Apr  6 18:30:18  ip6_tables
Apr  6 18:30:18  xt_state
Apr  6 18:30:18  nf_conntrack
Apr  6 18:30:18  sunrpc
Apr  6 18:30:18  usblp
Apr  6 18:30:18  snd_hda_codec_realtek
Apr  6 18:30:18  snd_hda_intel
Apr  6 18:30:18  snd_hda_codec
Apr  6 18:30:18  snd_hwdep
Apr  6 18:30:18  snd_seq
Apr  6 18:30:18  snd_seq_device
Apr  6 18:30:18  snd_pcm
Apr  6 18:30:18  i2c_i801
Apr  6 18:30:18  snd_timer
Apr  6 18:30:18  snd
Apr  6 18:30:18  soundcore
Apr  6 18:30:18  snd_page_alloc
Apr  6 18:30:18  iTCO_wdt
Apr  6 18:30:18  i2c_core
Apr  6 18:30:18  serio_raw
Apr  6 18:30:18  iTCO_vendor_support
Apr  6 18:30:18  e1000e
Apr  6 18:30:18  microcode
Apr  6 18:30:18  dcdbas
Apr  6 18:30:18  uinput
Apr  6 18:30:18  usb_storage
Apr  6 18:30:18  [last unloaded: netconsole]
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186011]
Apr  6 18:30:18 dellix [ 5358.186011] Pid: 16803, comm: kworker/1:224 Tainted: P           O 3.2.10-6969.fc16.x86_64 #1
Apr  6 18:30:18  Dell Inc. Inspiron 530
Apr  6 18:30:18 dellix /0RY007
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186011] RIP: 0010:[<ffffffffa0e50968>]
Apr  6 18:30:18  [<ffffffffa0e50968>] inode_go_sync+0x108/0x140 [gfs2]
Apr  6 18:30:18 dellix [ 5358.186011] RSP: 0018:ffff8800292a9ce0  EFLAGS: 00010202
Apr  6 18:30:18 dellix [ 5358.186011] RAX: 0000000000000003 RBX: ffff8800056a8b68 RCX: 0000000000000003
Apr  6 18:30:18 dellix [ 5358.186011] RDX: ffffffffa0e6d380 RSI: 0000000000000000 RDI: ffff8800056a8b68
Apr  6 18:30:18 dellix [ 5358.186011] RBP: ffff8800292a9d00 R08: ffff8800056a8c68 R09: ffff880010531310
Apr  6 18:30:18 dellix [ 5358.186011] R10: 000000000062765f R11: 0000000000000000 R12: 0000000000000000
Apr  6 18:30:18 dellix [ 5358.186011] R13: ffff8800056a8d00 R14: ffff8800056a8bb0 R15: ffffe8ffffc84305
Apr  6 18:30:18 dellix [ 5358.186011] FS:  0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
Apr  6 18:30:18 dellix [ 5358.186011] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  6 18:30:18 dellix [ 5358.186011] CR2: 00000000f0c9c000 CR3: 0000000048420000 CR4: 00000000000006e0
Apr  6 18:30:18 dellix [ 5358.186011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr  6 18:30:18 dellix [ 5358.186011] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr  6 18:30:18 dellix [ 5358.186011] Process kworker/1:224 (pid: 16803, threadinfo ffff8800292a8000, task ffff880040399730)
Apr  6 18:30:18 dellix [ 5358.186011] Stack:
Apr  6 18:30:18 dellix [ 5358.186011]  ffff880000000000
Apr  6 18:30:18  ffff8800056a8b68
Apr  6 18:30:18  ffff8800056a8bb0
Apr  6 18:30:18  ffff8800056a8bb0
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186011]  ffff8800292a9d70
Apr  6 18:30:18  ffffffffa0e4e9e9
Apr  6 18:30:18  ffff880040399778
Apr  6 18:30:18  00ff8800a5302e60
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186011]  ffff88005a190000
Apr  6 18:30:18  ffffffffa0e6d380
Apr  6 18:30:18  ffff8800056a8bb0
Apr  6 18:30:18  0000000000000000
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186011] Call Trace:
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffffa0e4e9e9>] do_xmote+0xa9/0x250 [gfs2]
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffffa0e4f1d8>] run_queue+0x148/0x2a0 [gfs2]
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff81065d7e>] ? try_to_wake_up+0x22e/0x290
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffffa0e4f39d>] glock_work_func+0x6d/0x150 [gfs2]
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffffa0e4f330>] ? run_queue+0x2a0/0x2a0 [gfs2]
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff8108a7ad>] process_one_work+0x11d/0x470
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff8108bbaf>] worker_thread+0x15f/0x350
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff8108ba50>] ? manage_workers+0x230/0x230
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff8109056c>] kthread+0x8c/0xa0
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff815eec34>] kernel_thread_helper+0x4/0x10
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff810904e0>] ? kthread_worker_fn+0x190/0x190
Apr  6 18:30:18 dellix [ 5358.186011]  [<ffffffff815eec30>] ? gs_change+0x13/0x13
Apr  6 18:30:18 dellix [ 5358.186011] Code:
Apr  6 18:30:18 dellix 85
Apr  6 18:30:18 dellix c0
Apr  6 18:30:18 dellix 0f
Apr  6 18:30:18 dellix 84
Apr  6 18:30:18 dellix 71
Apr  6 18:30:18 dellix ff
Apr  6 18:30:18 dellix ff
Apr  6 18:30:18 dellix ff
Apr  6 18:30:18 dellix 49
Apr  6 18:30:18 dellix 8b
Apr  6 18:30:18 dellix 7c
Apr  6 18:30:18 dellix 24
Apr  6 18:30:18 dellix 30
Apr  6 18:30:18 dellix 31
Apr  6 18:30:18 dellix c9
Apr  6 18:30:18 dellix 31
Apr  6 18:30:18 dellix d2
Apr  6 18:30:18 dellix 31
Apr  6 18:30:18 dellix f6
Apr  6 18:30:18 dellix e8
Apr  6 18:30:18 dellix 3b
Apr  6 18:30:18 dellix ee
Apr  6 18:30:18 dellix 2e
Apr  6 18:30:18 dellix e0
Apr  6 18:30:18 dellix e9
Apr  6 18:30:18 dellix 5c
Apr  6 18:30:18 dellix ff
Apr  6 18:30:18 dellix ff
Apr  6 18:30:18 dellix ff
Apr  6 18:30:18 dellix 83
Apr  6 18:30:18 dellix f8
Apr  6 18:30:18 dellix e4
Apr  6 18:30:18 dellix 74
Apr  6 18:30:18 dellix 23
Apr  6 18:30:18 dellix f0
Apr  6 18:30:18 dellix 41
Apr  6 18:30:18 dellix 80
Apr  6 18:30:18 dellix 4c
Apr  6 18:30:18 dellix 24
Apr  6 18:30:18 dellix 7b
Apr  6 18:30:18 dellix 01
Apr  6 18:30:18 dellix eb
Apr  6 18:30:18 dellix 9e
Apr  6 18:30:18 dellix f>
Apr  6 18:30:18 dellix 0b
Apr  6 18:30:18 dellix 83
Apr  6 18:30:18 dellix f8
Apr  6 18:30:18 dellix e4
Apr  6 18:30:18 dellix 0f
Apr  6 18:30:18 dellix 1f
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 74
Apr  6 18:30:18 dellix 1d
Apr  6 18:30:18 dellix f0
Apr  6 18:30:18 dellix 41
Apr  6 18:30:18 dellix 80
Apr  6 18:30:18 dellix 4d
Apr  6 18:30:18 dellix 7b
Apr  6 18:30:18 dellix 01
Apr  6 18:30:18 dellix 0f
Apr  6 18:30:18 dellix 1f
Apr  6 18:30:18 dellix 84
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186011] RIP
Apr  6 18:30:18  [<ffffffffa0e50968>] inode_go_sync+0x108/0x140 [gfs2]
Apr  6 18:30:18 dellix [ 5358.186011]  RSP <ffff8800292a9ce0>
Apr  6 18:30:18 dellix [ 5358.186523] ---[ end trace a8655a0250dbfd40 ]---
Apr  6 18:30:18 dellix [ 5358.186586] BUG: unable to handle kernel
Apr  6 18:30:18 paging request
Apr  6 18:30:18  at fffffffffffffff8
Apr  6 18:30:18 dellix [ 5358.186598] IP:
Apr  6 18:30:18  [<ffffffff81090841>] kthread_data+0x11/0x20
Apr  6 18:30:18 dellix [ 5358.186609] PGD 1a07067
Apr  6 18:30:18 PUD 1a08067
Apr  6 18:30:18 PMD 0
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186622] Oops: 0000 [#2]
Apr  6 18:30:18 dellix SMP
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186632] CPU 1
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186636] Modules linked in:
Apr  6 18:30:18  binfmt_misc
Apr  6 18:30:18  gfs2
Apr  6 18:30:18  dlm
Apr  6 18:30:18  sha256_generic
Apr  6 18:30:18  dm_crypt
Apr  6 18:30:18  netconsole
Apr  6 18:30:18  nvidia(P)
Apr  6 18:30:18  ebtable_nat
Apr  6 18:30:18  ebtables
Apr  6 18:30:18  ipt_MASQUERADE
Apr  6 18:30:18  iptable_nat
Apr  6 18:30:18  nf_nat
Apr  6 18:30:18  xt_CHECKSUM
Apr  6 18:30:18  iptable_mangle
Apr  6 18:30:18  tun
Apr  6 18:30:18  bridge
Apr  6 18:30:18  stp
Apr  6 18:30:18  llc
Apr  6 18:30:18  drbd
Apr  6 18:30:18  lockd
Apr  6 18:30:18  lru_cache
Apr  6 18:30:18  ip6t_REJECT
Apr  6 18:30:18  nf_defrag_ipv6
Apr  6 18:30:18  nf_conntrack_ipv4
Apr  6 18:30:18  ip6table_filter
Apr  6 18:30:18  nf_defrag_ipv4
Apr  6 18:30:18  nf_conntrack
Apr  6 18:30:18  sunrpc
Apr  6 18:30:18  usblp
Apr  6 18:30:18  snd_hda_codec_realtek
Apr  6 18:30:18  snd_hda_intel
Apr  6 18:30:18  snd_hda_codec
Apr  6 18:30:18  snd_hwdep
Apr  6 18:30:18  snd_seq
Apr  6 18:30:18  snd_seq_device
Apr  6 18:30:18  snd_pcm
Apr  6 18:30:18  i2c_i801
Apr  6 18:30:18  snd_timer
Apr  6 18:30:18  snd
Apr  6 18:30:18  soundcore
Apr  6 18:30:18  snd_page_alloc
Apr  6 18:30:18  i2c_core
Apr  6 18:30:18  serio_raw
Apr  6 18:30:18  iTCO_vendor_support
Apr  6 18:30:18  e1000e
Apr  6 18:30:18  dcdbas
Apr  6 18:30:18  uinput
Apr  6 18:30:18  usb_storage
Apr  6 18:30:18  [last unloaded: netconsole]
Apr  6 18:30:18 dellix [ 5358.186780] Pid: 16803, comm: kworker/1:224 Tainted: P      D    O 3.2.10-6969.fc16.x86_64 #1
Apr  6 18:30:18  Dell Inc. Inspiron 530
Apr  6 18:30:18 dellix /0RY007
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.186818] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff880040399730
Apr  6 18:30:18 dellix [ 5358.186824] RBP: ffff8800292a9998 R08: ffff8800403997a0 R09: 0000000000000000
Apr  6 18:30:18 dellix [ 5358.186836] R13: 0000000000000001 R14: 0000000000000001 R15: ffff880040399a00
Apr  6 18:30:18 dellix [ 5358.186854] CR2: fffffffffffffff8 CR3: 0000000048420000 CR4: 00000000000006e0
Apr  6 18:30:18 dellix [ 5358.186873] Process kworker/1:224 (pid: 16803, threadinfo ffff8800292a8000, task ffff880040399730)
Apr  6 18:30:18 dellix [ 5358.186879] Stack:
Apr  6 18:30:18  ffff8800292a9998
Apr  6 18:30:18  ffff8800cfc93780
Apr  6 18:30:18  ffff8800292a9a28
Apr  6 18:30:18 dellix
Apr  6 18:30:18  ffff8800292a97a0
Apr  6 18:30:18  ffff880040399a00
Apr  6 18:30:18  ffff880040399730
Apr  6 18:30:18 dellix [ 5358.186915]  ffff8800292a9fd8
Apr  6 18:30:18  ffff8800292a9fd8
Apr  6 18:30:18  ffff8800292a9fd8
Apr  6 18:30:18 dellix [ 5358.186930] Call Trace:
Apr  6 18:30:18 dellix [ 5358.186973]  [<ffffffff815e5c3b>] oops_end+0xab/0xf0
Apr  6 18:30:18 dellix [ 5358.186991]  [<ffffffff81014f05>] do_invalid_op+0x95/0xb0
Apr  6 18:30:18 dellix [ 5358.187004]  [<ffffffffa0e50968>] ? inode_go_sync+0x108/0x140 [gfs2]
Apr  6 18:30:18 dellix [ 5358.187017]  [<ffffffffa0e174ef>] ? __put_lkb+0x3f/0xd0 [dlm]
Apr  6 18:30:18 dellix [ 5358.187028]  [<ffffffffa0e1e55d>] ? dlm_put_lockspace+0x1d/0x30 [dlm]
Apr  6 18:30:18 dellix [ 5358.187062]  [<ffffffffa0e4e9e9>] do_xmote+0xa9/0x250 [gfs2]
Apr  6 18:30:18 dellix [ 5358.187075]  [<ffffffffa0e4f1d8>] run_queue+0x148/0x2a0 [gfs2]
Apr  6 18:30:18 dellix [ 5358.187082]  [<ffffffff81065d7e>] ? try_to_wake_up+0x22e/0x290
Apr  6 18:30:18 dellix [ 5358.187095]  [<ffffffffa0e4f39d>] glock_work_func+0x6d/0x150 [gfs2]
Apr  6 18:30:18 dellix [ 5358.187107]  [<ffffffffa0e4f330>] ? run_queue+0x2a0/0x2a0 [gfs2]
Apr  6 18:30:18 dellix [ 5358.187114]  [<ffffffff8108a7ad>] process_one_work+0x11d/0x470
Apr  6 18:30:18 dellix [ 5358.187121]  [<ffffffff8108bbaf>] worker_thread+0x15f/0x350
Apr  6 18:30:18 dellix [ 5358.187128]  [<ffffffff8108ba50>] ? manage_workers+0x230/0x230
Apr  6 18:30:18 dellix [ 5358.187135]  [<ffffffff8109056c>] kthread+0x8c/0xa0
Apr  6 18:30:18 dellix [ 5358.187143]  [<ffffffff815eec34>] kernel_thread_helper+0x4/0x10
Apr  6 18:30:18 dellix [ 5358.187150]  [<ffffffff810904e0>] ? kthread_worker_fn+0x190/0x190
Apr  6 18:30:18 dellix c7
Apr  6 18:30:18 dellix c7
Apr  6 18:30:18 dellix 18
Apr  6 18:30:18 dellix 7b
Apr  6 18:30:18 dellix 81
Apr  6 18:30:18 dellix e8
Apr  6 18:30:18 dellix 4f
Apr  6 18:30:18 dellix fd
Apr  6 18:30:18 dellix ff
Apr  6 18:30:18 dellix 48
Apr  6 18:30:18 dellix 5d
Apr  6 18:30:18 dellix f0
Apr  6 18:30:18 dellix 4c
Apr  6 18:30:18 dellix 8b
Apr  6 18:30:18 dellix 65
Apr  6 18:30:18 dellix f8
Apr  6 18:30:18 dellix c9
Apr  6 18:30:18 dellix c3
Apr  6 18:30:18 dellix 0f
Apr  6 18:30:18 dellix 1f
Apr  6 18:30:18 dellix 44
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 55
Apr  6 18:30:18 dellix 48
Apr  6 18:30:18 dellix 89
Apr  6 18:30:18 dellix e5
Apr  6 18:30:18 dellix 66
Apr  6 18:30:18 dellix 66
Apr  6 18:30:18 dellix 66
Apr  6 18:30:18 dellix 66
Apr  6 18:30:18 dellix 90
Apr  6 18:30:18 dellix 48
Apr  6 18:30:18 dellix 8b
Apr  6 18:30:18 dellix 87
Apr  6 18:30:18 dellix 70
Apr  6 18:30:18 dellix 03
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 5d
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix 8b
Apr  6 18:30:18 dellix 40
Apr  6 18:30:18 dellix f8
Apr  6 18:30:18 dellix c3
Apr  6 18:30:18 dellix 66
Apr  6 18:30:18 dellix 2e
Apr  6 18:30:18 dellix 0f
Apr  6 18:30:18 dellix 1f
Apr  6 18:30:18 dellix 84
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 00
Apr  6 18:30:18 dellix 48
Apr  6 18:30:18 dellix 89
Apr  6 18:30:18 dellix e5
Apr  6 18:30:18 dellix 66
Apr  6 18:30:18 dellix
Apr  6 18:30:18 dellix [ 5358.187285] RIP
Apr  6 18:30:18 dellix [ 5358.187292]  RSP <ffff8800292a9980>
Apr  6 18:30:18 dellix [ 5358.187296] CR2: fffffffffffffff8
Apr  6 18:30:52 dellix [ 5358.187516] ------------[ cut here ]------------
Apr  6 18:30:52 dellix [ 5358.187516] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0()
Apr  6 18:30:52 dellix [ 5358.187516] Hardware name: Inspiron 530
Apr  6 18:30:52 dellix [ 5358.187516] Watchdog detected hard LOCKUP on cpu 1
Apr  6 18:30:52 dellix
Apr  6 18:30:52 dellix [ 5358.187516] Modules linked in:
Apr  6 18:30:52  binfmt_misc
Apr  6 18:30:52  gfs2
Apr  6 18:30:52  dlm
Apr  6 18:30:52  sha256_generic
Apr  6 18:30:52  dm_crypt
Apr  6 18:30:52  netconsole
Apr  6 18:30:52  nvidia(P)
Apr  6 18:30:52  ebtable_nat
Apr  6 18:30:52  ebtables
Apr  6 18:30:52  ipt_MASQUERADE
Apr  6 18:30:52  iptable_nat
Apr  6 18:30:52  nf_nat
Apr  6 18:30:52  xt_CHECKSUM
Apr  6 18:30:52  iptable_mangle
Apr  6 18:30:52  tun
Apr  6 18:30:52  bridge
Apr  6 18:30:52  stp
Apr  6 18:30:52  llc
Apr  6 18:30:52  drbd
Apr  6 18:30:52  lockd
Apr  6 18:30:52  lru_cache
Apr  6 18:30:52  ip6t_REJECT
Apr  6 18:30:52  nf_conntrack_ipv6
Apr  6 18:30:52  nf_defrag_ipv6
Apr  6 18:30:52  nf_conntrack_ipv4
Apr  6 18:30:52  ip6table_filter
Apr  6 18:30:52  nf_defrag_ipv4
Apr  6 18:30:52  ip6_tables
Apr  6 18:30:52  xt_state
Apr  6 18:30:52  nf_conntrack
Apr  6 18:30:52  sunrpc
Apr  6 18:30:52  usblp
Apr  6 18:30:52  snd_hda_codec_realtek
Apr  6 18:30:52  snd_hda_intel
Apr  6 18:30:52  snd_hda_codec
Apr  6 18:30:52  snd_hwdep
Apr  6 18:30:52  snd_seq
Apr  6 18:30:52  snd_seq_device
Apr  6 18:30:52  snd_pcm
Apr  6 18:30:52  i2c_i801
Apr  6 18:30:52  snd_timer
Apr  6 18:30:52  snd
Apr  6 18:30:52  soundcore
Apr  6 18:30:52  snd_page_alloc
Apr  6 18:30:52  iTCO_wdt
Apr  6 18:30:52  i2c_core
Apr  6 18:30:52  serio_raw
Apr  6 18:30:52  iTCO_vendor_support
Apr  6 18:30:52  e1000e
Apr  6 18:30:52  microcode
Apr  6 18:30:52  dcdbas
Apr  6 18:30:52  uinput
Apr  6 18:30:52  usb_storage
Apr  6 18:30:52  [last unloaded: netconsole]
Apr  6 18:30:52 dellix
Apr  6 18:30:52 dellix [ 5358.187516] Pid: 16803, comm: kworker/1:224 Tainted: P      D    O 3.2.10-6969.fc16.x86_64 #1
Apr  6 18:30:52 dellix [ 5358.187516] Call Trace:
Apr  6 18:30:52 dellix [ 5358.187516]  <NMI>
Apr  6 18:30:52  [<ffffffff8106e72f>] warn_slowpath_common+0x7f/0xc0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8106e826>] warn_slowpath_fmt+0x46/0x50
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff810ddfba>] watchdog_overflow_callback+0x9a/0xc0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff811144e6>] __perf_event_overflow+0x96/0x1e0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81111ec1>] ? perf_event_update_userpage+0x11/0xc0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8102486a>] ? x86_perf_event_set_period+0xda/0x150
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81114e94>] perf_event_overflow+0x14/0x20
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81028dc3>] intel_pmu_handle_irq+0x163/0x2e0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e6591>] perf_event_nmi_handler+0x21/0x30
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e5e6a>] do_nmi+0xfa/0x350
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e54b0>] nmi+0x20/0x30
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e2bef>] ? schedule+0x3f/0x60
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e4dd0>] ? _raw_spin_lock_irq+0x30/0x40
Apr  6 18:30:52 dellix [ 5358.187516]  <<EOE>>
Apr  6 18:30:52  [<ffffffff815e20d1>] __schedule+0xc1/0x8c0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e2bef>] schedule+0x3f/0x60
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81072e52>] do_exit+0x792/0x8a0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8107045a>] ? kmsg_dump+0x4a/0xe0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e5c3b>] oops_end+0xab/0xf0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815d990f>] no_context+0x214/0x223
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8101ad52>] ? nommu_map_page+0x42/0x80
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815d9ae7>] __bad_area_nosemaphore+0x1c9/0x1e8
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815d91bb>] ? pmd_offset+0x1f/0x25
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815d9b19>] bad_area_nosemaphore+0x13/0x15
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e8726>] do_page_fault+0x416/0x4f0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff812c3e4b>] ? vsnprintf+0x42b/0x5d0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff811944e4>] ? evict+0x104/0x1a0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e51f5>] page_fault+0x25/0x30
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81090841>] ? kthread_data+0x11/0x20
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8108c2a5>] ? wq_worker_sleeping+0x15/0xa0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e25f7>] __schedule+0x5e7/0x8c0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e2bef>] schedule+0x3f/0x60
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81072c6c>] do_exit+0x5ac/0x8a0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8107045a>] ? kmsg_dump+0x4a/0xe0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e5c3b>] oops_end+0xab/0xf0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81017868>] die+0x58/0x90
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815e5594>] do_trap+0xc4/0x170
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81014f05>] do_invalid_op+0x95/0xb0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e50968>] ? inode_go_sync+0x108/0x140 [gfs2]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e174ef>] ? __put_lkb+0x3f/0xd0 [dlm]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e1e55d>] ? dlm_put_lockspace+0x1d/0x30 [dlm]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815eeaab>] invalid_op+0x1b/0x20
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e50968>] ? inode_go_sync+0x108/0x140 [gfs2]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e4e9e9>] do_xmote+0xa9/0x250 [gfs2]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e4f1d8>] run_queue+0x148/0x2a0 [gfs2]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff81065d7e>] ? try_to_wake_up+0x22e/0x290
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e4f39d>] glock_work_func+0x6d/0x150 [gfs2]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffffa0e4f330>] ? run_queue+0x2a0/0x2a0 [gfs2]
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8108a7ad>] process_one_work+0x11d/0x470
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8108bbaf>] worker_thread+0x15f/0x350
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8108ba50>] ? manage_workers+0x230/0x230
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff8109056c>] kthread+0x8c/0xa0
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815eec34>] kernel_thread_helper+0x4/0x10
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff810904e0>] ? kthread_worker_fn+0x190/0x190
Apr  6 18:30:52 dellix [ 5358.187516]  [<ffffffff815eec30>] ? gs_change+0x13/0x13
Apr  6 18:30:52 dellix [ 5358.187516] ---[ end trace a8655a0250dbfd42 ]---
Apr  6 18:31:09 dellix [ 5358.188010] ------------[ cut here ]------------
Apr  6 18:31:09 dellix [ 5358.188010] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0()
Apr  6 18:31:09 dellix [ 5358.188010] Hardware name: Inspiron 530
Apr  6 18:31:09 dellix [ 5358.188010] Watchdog detected hard LOCKUP on cpu 0
Apr  6 18:31:09 dellix
Apr  6 18:31:09 dellix [ 5358.188010] Modules linked in:
Apr  6 18:31:09  binfmt_misc
Apr  6 18:31:09  gfs2
Apr  6 18:31:09  dlm
Apr  6 18:31:09  sha256_generic
Apr  6 18:31:09  dm_crypt
Apr  6 18:31:09  netconsole
Apr  6 18:31:09  nvidia(P)
Apr  6 18:31:09  ebtable_nat
Apr  6 18:31:09  ebtables
Apr  6 18:31:09  ipt_MASQUERADE
Apr  6 18:31:09  iptable_nat
Apr  6 18:31:09  nf_nat
Apr  6 18:31:09  xt_CHECKSUM
Apr  6 18:31:09  iptable_mangle
Apr  6 18:31:09  tun
Apr  6 18:31:09  bridge
Apr  6 18:31:09  stp
Apr  6 18:31:09  llc
Apr  6 18:31:09  drbd
Apr  6 18:31:09  lockd
Apr  6 18:31:09  lru_cache
Apr  6 18:31:09  ip6t_REJECT
Apr  6 18:31:09  nf_conntrack_ipv6
Apr  6 18:31:09  nf_defrag_ipv6
Apr  6 18:31:09  nf_conntrack_ipv4
Apr  6 18:31:09  ip6table_filter
Apr  6 18:31:09  nf_defrag_ipv4
Apr  6 18:31:09  ip6_tables
Apr  6 18:31:09  xt_state
Apr  6 18:31:09  nf_conntrack
Apr  6 18:31:09  sunrpc
Apr  6 18:31:09  usblp
Apr  6 18:31:09  snd_hda_codec_realtek
Apr  6 18:31:09  snd_hda_intel
Apr  6 18:31:09  snd_hda_codec
Apr  6 18:31:09  snd_hwdep
Apr  6 18:31:09  snd_seq
Apr  6 18:31:09  snd_seq_device
Apr  6 18:31:09  snd_pcm
Apr  6 18:31:09  i2c_i801
Apr  6 18:31:09  snd_timer
Apr  6 18:31:09  snd
Apr  6 18:31:09  soundcore
Apr  6 18:31:09  snd_page_alloc
Apr  6 18:31:09  iTCO_wdt
Apr  6 18:31:09  i2c_core
Apr  6 18:31:09  serio_raw
Apr  6 18:31:09  iTCO_vendor_support
Apr  6 18:31:09  e1000e
Apr  6 18:31:09  microcode
Apr  6 18:31:09  dcdbas
Apr  6 18:31:09  uinput
Apr  6 18:31:09  usb_storage
Apr  6 18:31:09  [last unloaded: netconsole]
Apr  6 18:31:09 dellix
Apr  6 18:31:09 dellix [ 5358.188010] Pid: 14512, comm: rsyslogd Tainted: P      D W  O 3.2.10-6969.fc16.x86_64 #1
Apr  6 18:31:09 dellix [ 5358.188010] Call Trace:
Apr  6 18:31:09 dellix [ 5358.188010]  <NMI>
Apr  6 18:31:09  [<ffffffff8106e72f>] warn_slowpath_common+0x7f/0xc0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8106e826>] warn_slowpath_fmt+0x46/0x50
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff810ddfba>] watchdog_overflow_callback+0x9a/0xc0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff811144e6>] __perf_event_overflow+0x96/0x1e0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81111ec1>] ? perf_event_update_userpage+0x11/0xc0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8102486a>] ? x86_perf_event_set_period+0xda/0x150
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81114e94>] perf_event_overflow+0x14/0x20
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81028dc3>] intel_pmu_handle_irq+0x163/0x2e0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e6591>] perf_event_nmi_handler+0x21/0x30
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e5e6a>] do_nmi+0xfa/0x350
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e54b0>] nmi+0x20/0x30
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e4d95>] ? _raw_spin_lock+0x25/0x30
Apr  6 18:31:09 dellix [ 5358.188010]  <<EOE>>
Apr  6 18:31:09  [<ffffffff81057325>] double_rq_lock+0x55/0x60
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81064dd1>] load_balance+0x251/0x850
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8101b9c9>] ? sched_clock+0x9/0x10
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81096ea8>] ? sched_clock_cpu+0xa8/0x110
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81056f53>] ? update_rq_clock+0x43/0x1a0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8105c98e>] ? update_shares+0xce/0x100
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e2875>] __schedule+0x865/0x8c0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8111a310>] ? sleep_on_page+0x20/0x20
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e2bef>] schedule+0x3f/0x60
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e2c9f>] io_schedule+0x8f/0xd0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8111a31e>] sleep_on_page_killable+0xe/0x40
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e34bf>] __wait_on_bit+0x5f/0x90
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e3461>] ? __wait_on_bit+0x1/0x90
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8111cbe0>] wait_on_page_bit_killable+0x80/0x90
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81090e60>] ? autoremove_wake_function+0x50/0x50
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8111cc96>] __lock_page_or_retry+0xa6/0xd0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8111d0b1>] filemap_fault+0x3f1/0x4a0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8113cecf>] __do_fault+0x6f/0x520
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff8113fec7>] handle_pte_fault+0xf7/0xac0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e23e4>] ? __schedule+0x3d4/0x8c0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81140c48>] handle_mm_fault+0x1f8/0x350
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e8452>] do_page_fault+0x142/0x4f0
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff81185d52>] ? path_put+0x22/0x30
Apr  6 18:31:09 dellix [ 5358.188010]  [<ffffffff815e51f5>] page_fault+0x25/0x30
Apr  6 18:31:09 dellix [ 5358.188010] ---[ end trace a8655a0250dbfd43 ]---
Comment 1 Steve Whitehouse 2012-04-10 05:24:54 EDT
It is tricky to tell, but I suspect that this bug may have already been fixed. It is possible that you've run into the bug which this patch fixes:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=35e478f42271673f79066a1ed008c6604621c6fe

I'm not familiar with unison, so I've no idea what the i/o pattern is likely to be. You may, depending on the access patterns, and the nodes on which you are running the various commands, be running into contention on one or more inodes. That would explain the poor performance and also increase the likelihood of hitting the bug I mentioned above.
Comment 2 Paulo Castro 2012-04-16 17:17:55 EDT
I can see it happening still in the exact some place after having rebuilt the module with the previous patch included.
Further to that, I can also trigger the issue with a gfs2 module running on kernel 3.3 built out of the latest changes in HEAD bfecc60d8f6715ec6b38aa29c4f5a3570415dae0

unison by itself doesn't seem to be the problem as I have managed to run it from begin to end without incidents.
But as soon as I turn on Jenkins, which every so often polls some git repos that exist in the fs the issue surfaces and the machines is brought down to a halt with a subsequent increase in noise triggered by what must be increases CPU activity.

I can't yet triggered at will, e.g. not yet sure what is it exactly that makes it happen.
It happens after a random amount of time after unison+jenkins are running.

I'm not familiar with the codebase but would appreciate any hints on extra debugging that could lead to a resolution or more clues...
Comment 3 Steve Whitehouse 2012-04-17 04:31:43 EDT
Hmm, lets go back to first principles then, I guess. I'm not familiar with Jenkins so I'm not at all sure what it is doing.

How many nodes do you have? How many of those are running any kind of load? What kind of load is this? You mentioned syncing the content of the gfs2 filesystem with a usb disk, but which end is which? (i.e. is the gfs2 end all read i/o or is it the target of this operation rather than the source, or are there things going in both directions?)

Are you using the noatime mount parameter? and if not could you add this since it is likely to reduce the load a fair bit if most of the i/o on the gfs2 filesystem is read only.

I have been running the postmark and fsx benchmarks all yesterday on the latest -nmw git tree and I've not seen anything like this at all, so I wonder if this Jenkins thing is doing something odd.

Are you using any non-default options, either on mount, or chattr +j or something like that?

The part of the code that has hit the problem is in the glock code and specifically that part which writes back dirty inode data to disk before the inode is released. This is a generally well tested code path since it runs very often during any normal filesystem load, and especially at umount time and when there is memory pressure or remote nodes requesting glocks.

Invalid opcode though is a very odd message to get, since the code should be read only, nothing should have been able to change it. I'd normally expect to see "NULL pointer dereference" or something like that, so I'm not really sure what to make of this one at the moment.
Comment 4 Paulo Castro 2012-04-17 19:43:37 EDT
You're absolutely right...

Context:

This is a two node setup running GFS2 on top of DRBD.
Currently, only one node has the FS mounted.
The other node is just there as a "backup".

Software:

unison is a GUI based bi-directional rsync.
In a nutshell it takes two endpoints and checks for differences.
When it is finished you have the choice of manually syncing the differences to one side or to the other.

The way I'm running it, and up to the point of the failure it is only doing reads as there is nothing to sync/write yet.

Jenkins is the former Hudson it's just a CI tool. What it does in this context is poll regularly for changes some paths in the GFS volume which have git repositories.

Addendum:

No writes are involved in this apart from potentially atime updates ?!

Triggering it:

Triggering seems to be accomplished by having something like unison imposing a bit of load whilst running some git queries against the repos that exist on GFS.
It's not yet conclusive...

More info:

This is how far I got on trying to get more info about what is going on.

--- rpmbuild/BUILD/kernel-3.3.fc16/vanilla-3.3/fs/gfs2/glops.c  2012-03-18 23:15:34.000000000 +0000
+++ rpmbuild/BUILD/kernel-3.3.fc16/linux-3.3.x86_64/fs/gfs2/glops.c     2012-04-15 18:06:10.455113558 +0100
@@ -197,6 +197,19 @@
        if (!test_and_clear_bit(GLF_DIRTY, &gl->gl_flags))
                return;

+        if (gl->gl_state != LM_ST_EXCLUSIVE) {
+
+          printk(KERN_CRIT "PEC DBG: BUG is ON");
+          // struct inode *inode = ip->i_inode;
+          if(ip) {
+            printk("PEC DBG ino %lu  mode 0%6.6o  lk %d  uid %d  gid %d"
+                   "  sz %lu  blks %lu  cnt %u\n",
+                   ip->i_inode.i_ino, ip->i_inode.i_mode, ip->i_inode.i_nlink, ip->i_inode.i_uid,
+                   ip->i_inode.i_gid, ip->i_inode.i_size, ip->i_inode.i_blocks,
+                   atomic_read(&ip->i_inode.i_count));
+          }
+          printk(KERN_CRIT "PEC DBG: end of printing inode");
+        }
        BUG_ON(gl->gl_state != LM_ST_EXCLUSIVE);

... and based on this snippet this what I've collected so far....

Apr 15 18:28:50 PEC DBG ino 66329  mode 0100444  lk 1  uid 1000  gid 1000  sz 9577  blks 32  cnt 0
Apr 16 22:05:49 PEC DBG ino 238990  mode 0100444  lk 1  uid 1000  gid 1000  sz 185  blks 8  cnt 1
Apr 17 20:26:23 PEC DBG ino 786388  mode 0100444  lk 1  uid 1000  gid 1000  sz 5160  blks 24  cnt 1
Apr 17 22:13:58 PEC DBG ino 238990  mode 0100444  lk 1  uid 1000  gid 1000  sz 185  blks 8  cnt 0
Apr 17 22:40:51 PEC DBG ino 262170  mode 0100444  lk 1  uid 1000  gid 1000  sz 20420  blks 48  cnt 0

Next:

I'll try mounting with the noatime and try to trigger it.
Comment 5 Steve Whitehouse 2012-04-18 11:14:40 EDT
Note that you shouldn't have to hack printk's into the source to see what is going on. There is a complete set of tracepoints for gfs2 which should allow you to see all the activity with respect to glocks, if required.

Knowing whether noatime makes a difference will be a very useful data point I think.
Comment 6 Paulo Castro 2012-04-18 16:16:29 EDT
noatime does make a difference. We haven't crashed yet.

I've managed to emulate the behaviour of Jenkins and I've been submitting the server to 5 times more load since last night and it's still kicking.
Comment 7 Dave Jones 2012-10-23 11:38:59 EDT
# Mass update to all open bugs.

Kernel 3.6.2-1.fc16 has just been pushed to updates.
This update is a significant rebase from the previous version.

Please retest with this kernel, and let us know if your problem has been fixed.

In the event that you have upgraded to a newer release and the bug you reported
is still present, please change the version field to the newest release you have
encountered the issue with.  Before doing so, please ensure you are testing the
latest kernel update in that release and attach any new and relevant information
you may have gathered.

If you are not the original bug reporter and you still experience this bug,
please file a new report, as it is possible that you may be seeing a
different problem. 
(Please don't clone this bug, a fresh bug referencing this bug in the comment is sufficient).
Comment 8 Paulo Castro 2012-10-25 07:02:52 EDT
Bug is still present with the most recent kernel...
The only way to prevent it from happening is to mount the FS with the 'noatime' option.

Version-Release number of selected component (if applicable):

gfs2-cluster-3.1.3-1.fc16.x86_64
gfs2-utils-3.1.3-1.fc16.x86_64
kernel-3.6.2-1.fc16.x86_64

netconsole dump:

Oct 25 11:23:52 dellix [ 846.769711] ------------[ cut here ]------------
Oct 25 11:23:52 dellix [ 846.770003] kernel BUG at fs/gfs2/glops.c:199!
Oct 25 11:23:52 dellix [ 846.770003] invalid opcode: 0000 [#1]
Oct 25 11:23:52 dellix SMP
Oct 25 11:23:52 dellix
Oct 25 11:23:52 dellix [ 846.770003] Modules linked in:
Oct 25 11:23:52  gfs2
Oct 25 11:23:52  sha256_generic
Oct 25 11:23:52  dm_crypt
Oct 25 11:23:52  dlm
Oct 25 11:23:52  sctp
Oct 25 11:23:52  libcrc32c
Oct 25 11:23:52  netconsole
Oct 25 11:23:52  ebtable_nat
Oct 25 11:23:52  ebtables
Oct 25 11:23:52  ipt_MASQUERADE
Oct 25 11:23:52  iptable_nat
Oct 25 11:23:52  nf_nat
Oct 25 11:23:52  xt_CHECKSUM
Oct 25 11:23:52  iptable_mangle
Oct 25 11:23:52  tun
Oct 25 11:23:52  bridge
Oct 25 11:23:52  stp
Oct 25 11:23:52  llc
Oct 25 11:23:52  drbd
Oct 25 11:23:52  lru_cache
Oct 25 11:23:52  ip6t_REJECT
Oct 25 11:23:52  nf_conntrack_ipv4
Oct 25 11:23:52  nf_conntrack_ipv6
Oct 25 11:23:52  nf_defrag_ipv4
Oct 25 11:23:52  nf_defrag_ipv6
Oct 25 11:23:52  xt_state
Oct 25 11:23:52  nf_conntrack
Oct 25 11:23:52  ip6table_filter
Oct 25 11:23:52  ip6_tables
Oct 25 11:23:52  hwmon_vid
Oct 25 11:23:52  binfmt_misc
Oct 25 11:23:52  snd_hda_codec_realtek
Oct 25 11:23:52  snd_hda_intel
Oct 25 11:23:52  snd_hda_codec
Oct 25 11:23:52  snd_hwdep
Oct 25 11:23:52  iTCO_wdt
Oct 25 11:23:52  snd_seq
Oct 25 11:23:52  snd_seq_device
Oct 25 11:23:52  snd_pcm
Oct 25 11:23:52  iTCO_vendor_support
Oct 25 11:23:52  snd_timer
Oct 25 11:23:52  snd
Oct 25 11:23:52  lpc_ich
Oct 25 11:23:52  serio_raw
Oct 25 11:23:52  coretemp
Oct 25 11:23:52  microcode
Oct 25 11:23:52  soundcore
Oct 25 11:23:52  i2c_i801
Oct 25 11:23:52  snd_page_alloc
Oct 25 11:23:52  dcdbas
Oct 25 11:23:52  e1000e
Oct 25 11:23:52  mfd_core
Oct 25 11:23:52  usb_storage
Oct 25 11:23:52  nouveau
Oct 25 11:23:52  ttm
Oct 25 11:23:52  drm_kms_helper
Oct 25 11:23:52  drm
Oct 25 11:23:53  i2c_algo_bit
Oct 25 11:23:53  i2c_core
Oct 25 11:23:53  mxm_wmi
Oct 25 11:23:53  video
Oct 25 11:23:53  wmi
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.770003] CPU 1
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.770003] Pid: 7299, comm: kworker/1:8H Not tainted 3.6.2-1.fc16.x86_64 #1 Dell Inc. Inspiron 530/0RY007
Oct 25 11:23:53 dellix [ 846.770003] RIP: 0010:[<ffffffffa0416cc8>]
Oct 25 11:23:53  [<ffffffffa0416cc8>] inode_go_sync+0x108/0x140 [gfs2]
Oct 25 11:23:53 dellix [ 846.770003] RSP: 0000:ffff88003cf43c90  EFLAGS: 00010202
Oct 25 11:23:53 dellix [ 846.770003] RAX: 0000000000000003 RBX: ffff88003a7a6970 RCX: 0000000000000003
Oct 25 11:23:53 dellix [ 846.770003] RDX: ffffffffa0436b80 RSI: 0000000000000000 RDI: ffff88003a7a6970
Oct 25 11:23:53 dellix [ 846.770003] RBP: ffff88003cf43cb0 R08: ffff88003a7a6a98 R09: ffff8800c90d8000
Oct 25 11:23:53 dellix [ 846.770003] R10: 000000000003907f R11: 0000000000000000 R12: ffff88003b8de760
Oct 25 11:23:53 dellix [ 846.770003] R13: ffff88003a7a6b30 R14: ffff88003a7a69c0 R15: ffffffffa04155c0
Oct 25 11:23:53 dellix [ 846.770003] FS:  0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
Oct 25 11:23:53 dellix [ 846.770003] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 25 11:23:53 dellix [ 846.770003] CR2: 00007f796830f040 CR3: 0000000076f77000 CR4: 00000000000007e0
Oct 25 11:23:53 dellix [ 846.770003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 25 11:23:53 dellix [ 846.770003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 25 11:23:53 dellix [ 846.770003] Process kworker/1:8H (pid: 7299, threadinfo ffff88003cf42000, task ffff8800274fdc80)
Oct 25 11:23:53 dellix [ 846.770003] Stack:
Oct 25 11:23:53 dellix [ 846.770003]  ffff88003cf1e270
Oct 25 11:23:53  ffff88003a7a6970
Oct 25 11:23:53  ffff88003a7a69c0
Oct 25 11:23:53  ffff88003a7a69c0
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.770003]  ffff88003cf43d20
Oct 25 11:23:53  ffffffffa0414c76
Oct 25 11:23:53  ffff8800274fdcc8
Oct 25 11:23:53  00ff8800cfc93db0
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.770003]  ffff88003bd49000
Oct 25 11:23:53  ffffffffa0436b80
Oct 25 11:23:53  ffff88003a7a69c0
Oct 25 11:23:53  0000000000000000
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.770003] Call Trace:
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffffa0414c76>] do_xmote+0xc6/0x270 [gfs2]
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffffa0415468>] run_queue+0x148/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffffa041460e>] ? gfs2_glock_put+0x4e/0x190 [gfs2]
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffffa041562d>] glock_work_func+0x6d/0x150 [gfs2]
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffff81079c7a>] process_one_work+0x13a/0x530
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffff8107b07e>] worker_thread+0x18e/0x510
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffff8107aef0>] ? manage_workers+0x310/0x310
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffff81080153>] kthread+0x93/0xa0
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffff81622244>] kernel_thread_helper+0x4/0x10
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffff810800c0>] ? flush_kthread_worker+0xb0/0xb0
Oct 25 11:23:53 dellix [ 846.770003]  [<ffffffff81622240>] ? gs_change+0x13/0x13
Oct 25 11:23:53 dellix [ 846.770003] Code:
Oct 25 11:23:53 dellix 85
Oct 25 11:23:53 dellix c0
Oct 25 11:23:53 dellix 0f
Oct 25 11:23:53 dellix 84
Oct 25 11:23:53 dellix 71
Oct 25 11:23:53 dellix ff
Oct 25 11:23:53 dellix ff
Oct 25 11:23:53 dellix ff
Oct 25 11:23:53 dellix 49
Oct 25 11:23:53 dellix 8b
Oct 25 11:23:53 dellix 7c
Oct 25 11:23:53 dellix 24
Oct 25 11:23:53 dellix 30
Oct 25 11:23:53 dellix 31
Oct 25 11:23:53 dellix c9
Oct 25 11:23:53 dellix 31
Oct 25 11:23:53 dellix d2
Oct 25 11:23:53 dellix 31
Oct 25 11:23:53 dellix f6
Oct 25 11:23:53 dellix e8
Oct 25 11:23:53 dellix 4b
Oct 25 11:23:53 dellix c2
Oct 25 11:23:53 dellix d3
Oct 25 11:23:53 dellix e0
Oct 25 11:23:53 dellix e9
Oct 25 11:23:53 dellix 5c
Oct 25 11:23:53 dellix ff
Oct 25 11:23:53 dellix ff
Oct 25 11:23:53 dellix ff
Oct 25 11:23:53 dellix 83
Oct 25 11:23:53 dellix f8
Oct 25 11:23:53 dellix e4
Oct 25 11:23:53 dellix 74
Oct 25 11:23:53 dellix 23
Oct 25 11:23:53 dellix f0
Oct 25 11:23:53 dellix 41
Oct 25 11:23:53 dellix 80
Oct 25 11:23:53 dellix 4c
Oct 25 11:23:53 dellix 24
Oct 25 11:23:53 dellix 7b
Oct 25 11:23:53 dellix 02
Oct 25 11:23:53 dellix eb
Oct 25 11:23:53 dellix 9b
Oct 25 11:23:53 dellix f>
Oct 25 11:23:53 dellix 0b
Oct 25 11:23:53 dellix 83
Oct 25 11:23:53 dellix f8
Oct 25 11:23:53 dellix e4
Oct 25 11:23:53 dellix 0f
Oct 25 11:23:53 dellix 1f
Oct 25 11:23:53 dellix 00
Oct 25 11:23:53 dellix 74
Oct 25 11:23:53 dellix 1d
Oct 25 11:23:53 dellix f0
Oct 25 11:23:53 dellix 41
Oct 25 11:23:53 dellix 80
Oct 25 11:23:53 dellix 4d
Oct 25 11:23:53 dellix 7b
Oct 25 11:23:53 dellix 02
Oct 25 11:23:53 dellix 0f
Oct 25 11:23:53 dellix 1f
Oct 25 11:23:53 dellix 84
Oct 25 11:23:53 dellix 00
Oct 25 11:23:53 dellix 00
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.795207] RIP
Oct 25 11:23:53  [<ffffffffa0416cc8>] inode_go_sync+0x108/0x140 [gfs2]
Oct 25 11:23:53 dellix [ 846.795207]  RSP <ffff88003cf43c90>
Oct 25 11:23:53 dellix [ 846.822877] ---[ end trace 279603713a1c265f ]---
Oct 25 11:23:53 dellix [ 846.823305] BUG: unable to handle kernel
Oct 25 11:23:53 paging request
Oct 25 11:23:53  at fffffffffffffff8
Oct 25 11:23:53 dellix [ 846.823705] IP:
Oct 25 11:23:53  [<ffffffff81080421>] kthread_data+0x11/0x20
Oct 25 11:23:53 dellix [ 846.824001] PGD 1c0d067
Oct 25 11:23:53 PUD 1c0e067
Oct 25 11:23:53 PMD 0
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.824001] Oops: 0000 [#2]
Oct 25 11:23:53 dellix SMP
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.824001] Modules linked in:
Oct 25 11:23:53  gfs2
Oct 25 11:23:53  sha256_generic
Oct 25 11:23:53  dm_crypt
Oct 25 11:23:53  dlm
Oct 25 11:23:53  sctp
Oct 25 11:23:53  libcrc32c
Oct 25 11:23:53  netconsole
Oct 25 11:23:53  ebtable_nat
Oct 25 11:23:53  ebtables
Oct 25 11:23:53  ipt_MASQUERADE
Oct 25 11:23:53  iptable_nat
Oct 25 11:23:53  nf_nat
Oct 25 11:23:53  xt_CHECKSUM
Oct 25 11:23:53  iptable_mangle
Oct 25 11:23:53  tun
Oct 25 11:23:53  bridge
Oct 25 11:23:53  stp
Oct 25 11:23:53  llc
Oct 25 11:23:53  drbd
Oct 25 11:23:53  lru_cache
Oct 25 11:23:53  ip6t_REJECT
Oct 25 11:23:53  nf_conntrack_ipv4
Oct 25 11:23:53  nf_conntrack_ipv6
Oct 25 11:23:53  nf_defrag_ipv4
Oct 25 11:23:53  nf_defrag_ipv6
Oct 25 11:23:53  xt_state
Oct 25 11:23:53  nf_conntrack
Oct 25 11:23:53  ip6table_filter
Oct 25 11:23:53  ip6_tables
Oct 25 11:23:53  hwmon_vid
Oct 25 11:23:53  binfmt_misc
Oct 25 11:23:53  snd_hda_codec_realtek
Oct 25 11:23:53  snd_hda_intel
Oct 25 11:23:53  snd_hda_codec
Oct 25 11:23:53  snd_hwdep
Oct 25 11:23:53  iTCO_wdt
Oct 25 11:23:53  snd_seq
Oct 25 11:23:53  snd_seq_device
Oct 25 11:23:53  snd_pcm
Oct 25 11:23:53  iTCO_vendor_support
Oct 25 11:23:53  snd_timer
Oct 25 11:23:53  snd
Oct 25 11:23:53  lpc_ich
Oct 25 11:23:53  serio_raw
Oct 25 11:23:53  coretemp
Oct 25 11:23:53  microcode
Oct 25 11:23:53  soundcore
Oct 25 11:23:53  i2c_i801
Oct 25 11:23:53  snd_page_alloc
Oct 25 11:23:53  dcdbas
Oct 25 11:23:53  e1000e
Oct 25 11:23:53  mfd_core
Oct 25 11:23:53  usb_storage
Oct 25 11:23:53  nouveau
Oct 25 11:23:53  ttm
Oct 25 11:23:53  drm_kms_helper
Oct 25 11:23:53  drm
Oct 25 11:23:53  i2c_algo_bit
Oct 25 11:23:53  i2c_core
Oct 25 11:23:53  mxm_wmi
Oct 25 11:23:53  video
Oct 25 11:23:53  wmi
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.824001] CPU 1
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.824001] Pid: 7299, comm: kworker/1:8H Tainted: G      D      3.6.2-1.fc16.x86_64 #1 Dell Inc. Inspiron 530/0RY007
Oct 25 11:23:53 dellix [ 846.824001] RIP: 0010:[<ffffffff81080421>]
Oct 25 11:23:53  [<ffffffff81080421>] kthread_data+0x11/0x20
Oct 25 11:23:53 dellix [ 846.824001] RSP: 0018:ffff88003cf43940  EFLAGS: 00010096
Oct 25 11:23:53 dellix [ 846.824001] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
Oct 25 11:23:53 dellix [ 846.824001] RDX: ffff8800cbc05000 RSI: 0000000000000001 RDI: ffff8800274fdc80
Oct 25 11:23:53 dellix [ 846.824001] RBP: ffff88003cf43958 R08: ffff8800274fdcf0 R09: 0000000000000001
Oct 25 11:23:53 dellix [ 846.824001] R10: 0000000000000000 R11: 000000000000002f R12: 0000000000000001
Oct 25 11:23:53 dellix [ 846.824001] R13: ffff8800274fe050 R14: ffff88003cf43748 R15: ffff8800274fdf58
Oct 25 11:23:53 dellix [ 846.824001] FS:  0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
Oct 25 11:23:53 dellix [ 846.824001] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 25 11:23:53 dellix [ 846.824001] CR2: fffffffffffffff8 CR3: 0000000069b83000 CR4: 00000000000007e0
Oct 25 11:23:53 dellix [ 846.824001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 25 11:23:53 dellix [ 846.824001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 25 11:23:53 dellix [ 846.824001] Process kworker/1:8H (pid: 7299, threadinfo ffff88003cf42000, task ffff8800274fdc80)
Oct 25 11:23:53 dellix [ 846.824001] Stack:
Oct 25 11:23:53 dellix [ 846.824001]  ffffffff8107b9a5
Oct 25 11:23:53  ffff88003cf43958
Oct 25 11:23:53  ffff8800cfc93d40
Oct 25 11:23:53  ffff88003cf439d8
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.824001]  ffffffff81617b2f
Oct 25 11:23:53  ffff88003cf43998
Oct 25 11:23:53  ffffffff812baf42
Oct 25 11:23:53  ffff8800274fdc80
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.824001]  ffff88003cf43fd8
Oct 25 11:23:53  ffff88003cf43fd8
Oct 25 11:23:53  ffff88003cf43fd8
Oct 25 11:23:53  ffff88003cf439c8
Oct 25 11:23:53 dellix
Oct 25 11:23:53 dellix [ 846.824001] Call Trace:
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff8107b9a5>] ? wq_worker_sleeping+0x15/0xc0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81617b2f>] __schedule+0x5df/0x7c0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff812baf42>] ? put_io_context+0x72/0xa0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81618019>] schedule+0x29/0x70
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff810628cd>] do_exit+0x58d/0x8d0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff8160e90a>] ? printk+0x61/0x63
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff8161a23c>] oops_end+0xac/0xf0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff810177f8>] die+0x58/0x90
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81619b14>] do_trap+0xc4/0x170
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81014e85>] do_invalid_op+0x95/0xb0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa0416cc8>] ? inode_go_sync+0x108/0x140 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa0432210>] ? gfs2_control_func+0x3e0/0x3e0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff811385b2>] ? pagevec_lookup+0x22/0x30
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa04317f0>] ? gdlm_lock+0x330/0x330 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa0432210>] ? gfs2_control_func+0x3e0/0x3e0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff816220bb>] invalid_op+0x1b/0x20
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa0416cc8>] ? inode_go_sync+0x108/0x140 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa0414c76>] do_xmote+0xc6/0x270 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa0415468>] run_queue+0x148/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa041460e>] ? gfs2_glock_put+0x4e/0x190 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffffa041562d>] glock_work_func+0x6d/0x150 [gfs2]
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81079c7a>] process_one_work+0x13a/0x530
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff8107b07e>] worker_thread+0x18e/0x510
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff8107aef0>] ? manage_workers+0x310/0x310
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81080153>] kthread+0x93/0xa0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81622244>] kernel_thread_helper+0x4/0x10
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff810800c0>] ? flush_kthread_worker+0xb0/0xb0
Oct 25 11:23:53 dellix [ 846.824001]  [<ffffffff81622240>] ? gs_change+0x13/0x13
Oct 25 11:23:53 dellix [ 846.824001] Code:
Oct 25 11:23:53 dellix 81
Oct 25 11:23:53 dellix e8
Oct 25 11:23:53 dellix e4
Oct 25 11:23:53 dellix c2
Oct 25 11:23:53 dellix fd
Oct 25 11:23:53 dellix ff
Oct 25 11:23:53 dellix e9
Comment 10 Steve Whitehouse 2012-11-08 09:09:35 EST
Fix also posted to the 3.6 stable tree.
Comment 11 Steve Whitehouse 2012-11-15 07:02:29 EST
Fix has been added to the 3.6 stable tree, so it should appear in the next stable tree release.
Comment 12 Steve Whitehouse 2012-11-27 05:01:24 EST
Now in the 3.6.8 stable tree

Note You need to log in before you can comment on or make changes to this bug.