Description of problem: Whilst trying to synchronise data across a GFS filesystem and a USB disk using unison there comes a point where the whole system grinds to a hard lock. Running an fsck.gfs2 on the FS doesn't seem to shed any light on anything particularly wrong. Using netconsole I was able to have access to the kernels last screams of horror... Version-Release number of selected component (if applicable): gfs2-cluster-3.1.3-1.fc16.x86_64 gfs2-utils-3.1.3-1.fc16.x86_64 kernel-3.2.10-3.fc16.x86_64 How reproducible: Every time. Steps to Reproduce: 1. Mount the GFS fs 2. Start unison synchronisation 3. Start something else that might access the GFS fs like a watcher for git repo changes Actual results: System hard lock. A couple of minutes past and I witness an increase in CPU activity by relating with the speed of the fans increasing. The system hands in such a violent way that no log ever gets written anywhere. The only way I could get it to spew something was by using netconsole. Expected results: I would expect none of this to happen :p Additional info: Please let me know if there's more info I can supply. Here follows the netconsole dump... Apr 6 18:30:18 dellix [ 5358.185950] ------------[ cut here ]------------ Apr 6 18:30:18 dellix [ 5358.185967] kernel BUG at fs/gfs2/glops.c:200! Apr 6 18:30:18 dellix [ 5358.185971] invalid opcode: 0000 [#1] Apr 6 18:30:18 dellix SMP Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.185979] CPU 1 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.185982] Modules linked in: Apr 6 18:30:18 binfmt_misc Apr 6 18:30:18 gfs2 Apr 6 18:30:18 dlm Apr 6 18:30:18 sha256_generic Apr 6 18:30:18 dm_crypt Apr 6 18:30:18 netconsole Apr 6 18:30:18 nvidia(P) Apr 6 18:30:18 ebtable_nat Apr 6 18:30:18 ebtables Apr 6 18:30:18 ipt_MASQUERADE Apr 6 18:30:18 iptable_nat Apr 6 18:30:18 nf_nat Apr 6 18:30:18 xt_CHECKSUM Apr 6 18:30:18 iptable_mangle Apr 6 18:30:18 tun Apr 6 18:30:18 bridge Apr 6 18:30:18 stp Apr 6 18:30:18 llc Apr 6 18:30:18 drbd Apr 6 18:30:18 lockd Apr 6 18:30:18 lru_cache Apr 6 18:30:18 ip6t_REJECT Apr 6 18:30:18 nf_conntrack_ipv6 Apr 6 18:30:18 nf_defrag_ipv6 Apr 6 18:30:18 nf_conntrack_ipv4 Apr 6 18:30:18 ip6table_filter Apr 6 18:30:18 nf_defrag_ipv4 Apr 6 18:30:18 ip6_tables Apr 6 18:30:18 xt_state Apr 6 18:30:18 nf_conntrack Apr 6 18:30:18 sunrpc Apr 6 18:30:18 usblp Apr 6 18:30:18 snd_hda_codec_realtek Apr 6 18:30:18 snd_hda_intel Apr 6 18:30:18 snd_hda_codec Apr 6 18:30:18 snd_hwdep Apr 6 18:30:18 snd_seq Apr 6 18:30:18 snd_seq_device Apr 6 18:30:18 snd_pcm Apr 6 18:30:18 i2c_i801 Apr 6 18:30:18 snd_timer Apr 6 18:30:18 snd Apr 6 18:30:18 soundcore Apr 6 18:30:18 snd_page_alloc Apr 6 18:30:18 iTCO_wdt Apr 6 18:30:18 i2c_core Apr 6 18:30:18 serio_raw Apr 6 18:30:18 iTCO_vendor_support Apr 6 18:30:18 e1000e Apr 6 18:30:18 microcode Apr 6 18:30:18 dcdbas Apr 6 18:30:18 uinput Apr 6 18:30:18 usb_storage Apr 6 18:30:18 [last unloaded: netconsole] Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186011] Apr 6 18:30:18 dellix [ 5358.186011] Pid: 16803, comm: kworker/1:224 Tainted: P O 3.2.10-6969.fc16.x86_64 #1 Apr 6 18:30:18 Dell Inc. Inspiron 530 Apr 6 18:30:18 dellix /0RY007 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186011] RIP: 0010:[<ffffffffa0e50968>] Apr 6 18:30:18 [<ffffffffa0e50968>] inode_go_sync+0x108/0x140 [gfs2] Apr 6 18:30:18 dellix [ 5358.186011] RSP: 0018:ffff8800292a9ce0 EFLAGS: 00010202 Apr 6 18:30:18 dellix [ 5358.186011] RAX: 0000000000000003 RBX: ffff8800056a8b68 RCX: 0000000000000003 Apr 6 18:30:18 dellix [ 5358.186011] RDX: ffffffffa0e6d380 RSI: 0000000000000000 RDI: ffff8800056a8b68 Apr 6 18:30:18 dellix [ 5358.186011] RBP: ffff8800292a9d00 R08: ffff8800056a8c68 R09: ffff880010531310 Apr 6 18:30:18 dellix [ 5358.186011] R10: 000000000062765f R11: 0000000000000000 R12: 0000000000000000 Apr 6 18:30:18 dellix [ 5358.186011] R13: ffff8800056a8d00 R14: ffff8800056a8bb0 R15: ffffe8ffffc84305 Apr 6 18:30:18 dellix [ 5358.186011] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000 Apr 6 18:30:18 dellix [ 5358.186011] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Apr 6 18:30:18 dellix [ 5358.186011] CR2: 00000000f0c9c000 CR3: 0000000048420000 CR4: 00000000000006e0 Apr 6 18:30:18 dellix [ 5358.186011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Apr 6 18:30:18 dellix [ 5358.186011] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Apr 6 18:30:18 dellix [ 5358.186011] Process kworker/1:224 (pid: 16803, threadinfo ffff8800292a8000, task ffff880040399730) Apr 6 18:30:18 dellix [ 5358.186011] Stack: Apr 6 18:30:18 dellix [ 5358.186011] ffff880000000000 Apr 6 18:30:18 ffff8800056a8b68 Apr 6 18:30:18 ffff8800056a8bb0 Apr 6 18:30:18 ffff8800056a8bb0 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186011] ffff8800292a9d70 Apr 6 18:30:18 ffffffffa0e4e9e9 Apr 6 18:30:18 ffff880040399778 Apr 6 18:30:18 00ff8800a5302e60 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186011] ffff88005a190000 Apr 6 18:30:18 ffffffffa0e6d380 Apr 6 18:30:18 ffff8800056a8bb0 Apr 6 18:30:18 0000000000000000 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186011] Call Trace: Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffffa0e4e9e9>] do_xmote+0xa9/0x250 [gfs2] Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffffa0e4f1d8>] run_queue+0x148/0x2a0 [gfs2] Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff81065d7e>] ? try_to_wake_up+0x22e/0x290 Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffffa0e4f39d>] glock_work_func+0x6d/0x150 [gfs2] Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffffa0e4f330>] ? run_queue+0x2a0/0x2a0 [gfs2] Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff8108a7ad>] process_one_work+0x11d/0x470 Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff8108bbaf>] worker_thread+0x15f/0x350 Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff8108ba50>] ? manage_workers+0x230/0x230 Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff8109056c>] kthread+0x8c/0xa0 Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff815eec34>] kernel_thread_helper+0x4/0x10 Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff810904e0>] ? kthread_worker_fn+0x190/0x190 Apr 6 18:30:18 dellix [ 5358.186011] [<ffffffff815eec30>] ? gs_change+0x13/0x13 Apr 6 18:30:18 dellix [ 5358.186011] Code: Apr 6 18:30:18 dellix 85 Apr 6 18:30:18 dellix c0 Apr 6 18:30:18 dellix 0f Apr 6 18:30:18 dellix 84 Apr 6 18:30:18 dellix 71 Apr 6 18:30:18 dellix ff Apr 6 18:30:18 dellix ff Apr 6 18:30:18 dellix ff Apr 6 18:30:18 dellix 49 Apr 6 18:30:18 dellix 8b Apr 6 18:30:18 dellix 7c Apr 6 18:30:18 dellix 24 Apr 6 18:30:18 dellix 30 Apr 6 18:30:18 dellix 31 Apr 6 18:30:18 dellix c9 Apr 6 18:30:18 dellix 31 Apr 6 18:30:18 dellix d2 Apr 6 18:30:18 dellix 31 Apr 6 18:30:18 dellix f6 Apr 6 18:30:18 dellix e8 Apr 6 18:30:18 dellix 3b Apr 6 18:30:18 dellix ee Apr 6 18:30:18 dellix 2e Apr 6 18:30:18 dellix e0 Apr 6 18:30:18 dellix e9 Apr 6 18:30:18 dellix 5c Apr 6 18:30:18 dellix ff Apr 6 18:30:18 dellix ff Apr 6 18:30:18 dellix ff Apr 6 18:30:18 dellix 83 Apr 6 18:30:18 dellix f8 Apr 6 18:30:18 dellix e4 Apr 6 18:30:18 dellix 74 Apr 6 18:30:18 dellix 23 Apr 6 18:30:18 dellix f0 Apr 6 18:30:18 dellix 41 Apr 6 18:30:18 dellix 80 Apr 6 18:30:18 dellix 4c Apr 6 18:30:18 dellix 24 Apr 6 18:30:18 dellix 7b Apr 6 18:30:18 dellix 01 Apr 6 18:30:18 dellix eb Apr 6 18:30:18 dellix 9e Apr 6 18:30:18 dellix f> Apr 6 18:30:18 dellix 0b Apr 6 18:30:18 dellix 83 Apr 6 18:30:18 dellix f8 Apr 6 18:30:18 dellix e4 Apr 6 18:30:18 dellix 0f Apr 6 18:30:18 dellix 1f Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 74 Apr 6 18:30:18 dellix 1d Apr 6 18:30:18 dellix f0 Apr 6 18:30:18 dellix 41 Apr 6 18:30:18 dellix 80 Apr 6 18:30:18 dellix 4d Apr 6 18:30:18 dellix 7b Apr 6 18:30:18 dellix 01 Apr 6 18:30:18 dellix 0f Apr 6 18:30:18 dellix 1f Apr 6 18:30:18 dellix 84 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186011] RIP Apr 6 18:30:18 [<ffffffffa0e50968>] inode_go_sync+0x108/0x140 [gfs2] Apr 6 18:30:18 dellix [ 5358.186011] RSP <ffff8800292a9ce0> Apr 6 18:30:18 dellix [ 5358.186523] ---[ end trace a8655a0250dbfd40 ]--- Apr 6 18:30:18 dellix [ 5358.186586] BUG: unable to handle kernel Apr 6 18:30:18 paging request Apr 6 18:30:18 at fffffffffffffff8 Apr 6 18:30:18 dellix [ 5358.186598] IP: Apr 6 18:30:18 [<ffffffff81090841>] kthread_data+0x11/0x20 Apr 6 18:30:18 dellix [ 5358.186609] PGD 1a07067 Apr 6 18:30:18 PUD 1a08067 Apr 6 18:30:18 PMD 0 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186622] Oops: 0000 [#2] Apr 6 18:30:18 dellix SMP Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186632] CPU 1 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186636] Modules linked in: Apr 6 18:30:18 binfmt_misc Apr 6 18:30:18 gfs2 Apr 6 18:30:18 dlm Apr 6 18:30:18 sha256_generic Apr 6 18:30:18 dm_crypt Apr 6 18:30:18 netconsole Apr 6 18:30:18 nvidia(P) Apr 6 18:30:18 ebtable_nat Apr 6 18:30:18 ebtables Apr 6 18:30:18 ipt_MASQUERADE Apr 6 18:30:18 iptable_nat Apr 6 18:30:18 nf_nat Apr 6 18:30:18 xt_CHECKSUM Apr 6 18:30:18 iptable_mangle Apr 6 18:30:18 tun Apr 6 18:30:18 bridge Apr 6 18:30:18 stp Apr 6 18:30:18 llc Apr 6 18:30:18 drbd Apr 6 18:30:18 lockd Apr 6 18:30:18 lru_cache Apr 6 18:30:18 ip6t_REJECT Apr 6 18:30:18 nf_defrag_ipv6 Apr 6 18:30:18 nf_conntrack_ipv4 Apr 6 18:30:18 ip6table_filter Apr 6 18:30:18 nf_defrag_ipv4 Apr 6 18:30:18 nf_conntrack Apr 6 18:30:18 sunrpc Apr 6 18:30:18 usblp Apr 6 18:30:18 snd_hda_codec_realtek Apr 6 18:30:18 snd_hda_intel Apr 6 18:30:18 snd_hda_codec Apr 6 18:30:18 snd_hwdep Apr 6 18:30:18 snd_seq Apr 6 18:30:18 snd_seq_device Apr 6 18:30:18 snd_pcm Apr 6 18:30:18 i2c_i801 Apr 6 18:30:18 snd_timer Apr 6 18:30:18 snd Apr 6 18:30:18 soundcore Apr 6 18:30:18 snd_page_alloc Apr 6 18:30:18 i2c_core Apr 6 18:30:18 serio_raw Apr 6 18:30:18 iTCO_vendor_support Apr 6 18:30:18 e1000e Apr 6 18:30:18 dcdbas Apr 6 18:30:18 uinput Apr 6 18:30:18 usb_storage Apr 6 18:30:18 [last unloaded: netconsole] Apr 6 18:30:18 dellix [ 5358.186780] Pid: 16803, comm: kworker/1:224 Tainted: P D O 3.2.10-6969.fc16.x86_64 #1 Apr 6 18:30:18 Dell Inc. Inspiron 530 Apr 6 18:30:18 dellix /0RY007 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.186818] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff880040399730 Apr 6 18:30:18 dellix [ 5358.186824] RBP: ffff8800292a9998 R08: ffff8800403997a0 R09: 0000000000000000 Apr 6 18:30:18 dellix [ 5358.186836] R13: 0000000000000001 R14: 0000000000000001 R15: ffff880040399a00 Apr 6 18:30:18 dellix [ 5358.186854] CR2: fffffffffffffff8 CR3: 0000000048420000 CR4: 00000000000006e0 Apr 6 18:30:18 dellix [ 5358.186873] Process kworker/1:224 (pid: 16803, threadinfo ffff8800292a8000, task ffff880040399730) Apr 6 18:30:18 dellix [ 5358.186879] Stack: Apr 6 18:30:18 ffff8800292a9998 Apr 6 18:30:18 ffff8800cfc93780 Apr 6 18:30:18 ffff8800292a9a28 Apr 6 18:30:18 dellix Apr 6 18:30:18 ffff8800292a97a0 Apr 6 18:30:18 ffff880040399a00 Apr 6 18:30:18 ffff880040399730 Apr 6 18:30:18 dellix [ 5358.186915] ffff8800292a9fd8 Apr 6 18:30:18 ffff8800292a9fd8 Apr 6 18:30:18 ffff8800292a9fd8 Apr 6 18:30:18 dellix [ 5358.186930] Call Trace: Apr 6 18:30:18 dellix [ 5358.186973] [<ffffffff815e5c3b>] oops_end+0xab/0xf0 Apr 6 18:30:18 dellix [ 5358.186991] [<ffffffff81014f05>] do_invalid_op+0x95/0xb0 Apr 6 18:30:18 dellix [ 5358.187004] [<ffffffffa0e50968>] ? inode_go_sync+0x108/0x140 [gfs2] Apr 6 18:30:18 dellix [ 5358.187017] [<ffffffffa0e174ef>] ? __put_lkb+0x3f/0xd0 [dlm] Apr 6 18:30:18 dellix [ 5358.187028] [<ffffffffa0e1e55d>] ? dlm_put_lockspace+0x1d/0x30 [dlm] Apr 6 18:30:18 dellix [ 5358.187062] [<ffffffffa0e4e9e9>] do_xmote+0xa9/0x250 [gfs2] Apr 6 18:30:18 dellix [ 5358.187075] [<ffffffffa0e4f1d8>] run_queue+0x148/0x2a0 [gfs2] Apr 6 18:30:18 dellix [ 5358.187082] [<ffffffff81065d7e>] ? try_to_wake_up+0x22e/0x290 Apr 6 18:30:18 dellix [ 5358.187095] [<ffffffffa0e4f39d>] glock_work_func+0x6d/0x150 [gfs2] Apr 6 18:30:18 dellix [ 5358.187107] [<ffffffffa0e4f330>] ? run_queue+0x2a0/0x2a0 [gfs2] Apr 6 18:30:18 dellix [ 5358.187114] [<ffffffff8108a7ad>] process_one_work+0x11d/0x470 Apr 6 18:30:18 dellix [ 5358.187121] [<ffffffff8108bbaf>] worker_thread+0x15f/0x350 Apr 6 18:30:18 dellix [ 5358.187128] [<ffffffff8108ba50>] ? manage_workers+0x230/0x230 Apr 6 18:30:18 dellix [ 5358.187135] [<ffffffff8109056c>] kthread+0x8c/0xa0 Apr 6 18:30:18 dellix [ 5358.187143] [<ffffffff815eec34>] kernel_thread_helper+0x4/0x10 Apr 6 18:30:18 dellix [ 5358.187150] [<ffffffff810904e0>] ? kthread_worker_fn+0x190/0x190 Apr 6 18:30:18 dellix c7 Apr 6 18:30:18 dellix c7 Apr 6 18:30:18 dellix 18 Apr 6 18:30:18 dellix 7b Apr 6 18:30:18 dellix 81 Apr 6 18:30:18 dellix e8 Apr 6 18:30:18 dellix 4f Apr 6 18:30:18 dellix fd Apr 6 18:30:18 dellix ff Apr 6 18:30:18 dellix 48 Apr 6 18:30:18 dellix 5d Apr 6 18:30:18 dellix f0 Apr 6 18:30:18 dellix 4c Apr 6 18:30:18 dellix 8b Apr 6 18:30:18 dellix 65 Apr 6 18:30:18 dellix f8 Apr 6 18:30:18 dellix c9 Apr 6 18:30:18 dellix c3 Apr 6 18:30:18 dellix 0f Apr 6 18:30:18 dellix 1f Apr 6 18:30:18 dellix 44 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 55 Apr 6 18:30:18 dellix 48 Apr 6 18:30:18 dellix 89 Apr 6 18:30:18 dellix e5 Apr 6 18:30:18 dellix 66 Apr 6 18:30:18 dellix 66 Apr 6 18:30:18 dellix 66 Apr 6 18:30:18 dellix 66 Apr 6 18:30:18 dellix 90 Apr 6 18:30:18 dellix 48 Apr 6 18:30:18 dellix 8b Apr 6 18:30:18 dellix 87 Apr 6 18:30:18 dellix 70 Apr 6 18:30:18 dellix 03 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 5d Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix 8b Apr 6 18:30:18 dellix 40 Apr 6 18:30:18 dellix f8 Apr 6 18:30:18 dellix c3 Apr 6 18:30:18 dellix 66 Apr 6 18:30:18 dellix 2e Apr 6 18:30:18 dellix 0f Apr 6 18:30:18 dellix 1f Apr 6 18:30:18 dellix 84 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 00 Apr 6 18:30:18 dellix 48 Apr 6 18:30:18 dellix 89 Apr 6 18:30:18 dellix e5 Apr 6 18:30:18 dellix 66 Apr 6 18:30:18 dellix Apr 6 18:30:18 dellix [ 5358.187285] RIP Apr 6 18:30:18 dellix [ 5358.187292] RSP <ffff8800292a9980> Apr 6 18:30:18 dellix [ 5358.187296] CR2: fffffffffffffff8 Apr 6 18:30:52 dellix [ 5358.187516] ------------[ cut here ]------------ Apr 6 18:30:52 dellix [ 5358.187516] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0() Apr 6 18:30:52 dellix [ 5358.187516] Hardware name: Inspiron 530 Apr 6 18:30:52 dellix [ 5358.187516] Watchdog detected hard LOCKUP on cpu 1 Apr 6 18:30:52 dellix Apr 6 18:30:52 dellix [ 5358.187516] Modules linked in: Apr 6 18:30:52 binfmt_misc Apr 6 18:30:52 gfs2 Apr 6 18:30:52 dlm Apr 6 18:30:52 sha256_generic Apr 6 18:30:52 dm_crypt Apr 6 18:30:52 netconsole Apr 6 18:30:52 nvidia(P) Apr 6 18:30:52 ebtable_nat Apr 6 18:30:52 ebtables Apr 6 18:30:52 ipt_MASQUERADE Apr 6 18:30:52 iptable_nat Apr 6 18:30:52 nf_nat Apr 6 18:30:52 xt_CHECKSUM Apr 6 18:30:52 iptable_mangle Apr 6 18:30:52 tun Apr 6 18:30:52 bridge Apr 6 18:30:52 stp Apr 6 18:30:52 llc Apr 6 18:30:52 drbd Apr 6 18:30:52 lockd Apr 6 18:30:52 lru_cache Apr 6 18:30:52 ip6t_REJECT Apr 6 18:30:52 nf_conntrack_ipv6 Apr 6 18:30:52 nf_defrag_ipv6 Apr 6 18:30:52 nf_conntrack_ipv4 Apr 6 18:30:52 ip6table_filter Apr 6 18:30:52 nf_defrag_ipv4 Apr 6 18:30:52 ip6_tables Apr 6 18:30:52 xt_state Apr 6 18:30:52 nf_conntrack Apr 6 18:30:52 sunrpc Apr 6 18:30:52 usblp Apr 6 18:30:52 snd_hda_codec_realtek Apr 6 18:30:52 snd_hda_intel Apr 6 18:30:52 snd_hda_codec Apr 6 18:30:52 snd_hwdep Apr 6 18:30:52 snd_seq Apr 6 18:30:52 snd_seq_device Apr 6 18:30:52 snd_pcm Apr 6 18:30:52 i2c_i801 Apr 6 18:30:52 snd_timer Apr 6 18:30:52 snd Apr 6 18:30:52 soundcore Apr 6 18:30:52 snd_page_alloc Apr 6 18:30:52 iTCO_wdt Apr 6 18:30:52 i2c_core Apr 6 18:30:52 serio_raw Apr 6 18:30:52 iTCO_vendor_support Apr 6 18:30:52 e1000e Apr 6 18:30:52 microcode Apr 6 18:30:52 dcdbas Apr 6 18:30:52 uinput Apr 6 18:30:52 usb_storage Apr 6 18:30:52 [last unloaded: netconsole] Apr 6 18:30:52 dellix Apr 6 18:30:52 dellix [ 5358.187516] Pid: 16803, comm: kworker/1:224 Tainted: P D O 3.2.10-6969.fc16.x86_64 #1 Apr 6 18:30:52 dellix [ 5358.187516] Call Trace: Apr 6 18:30:52 dellix [ 5358.187516] <NMI> Apr 6 18:30:52 [<ffffffff8106e72f>] warn_slowpath_common+0x7f/0xc0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8106e826>] warn_slowpath_fmt+0x46/0x50 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff810ddfba>] watchdog_overflow_callback+0x9a/0xc0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff811144e6>] __perf_event_overflow+0x96/0x1e0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81111ec1>] ? perf_event_update_userpage+0x11/0xc0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8102486a>] ? x86_perf_event_set_period+0xda/0x150 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81114e94>] perf_event_overflow+0x14/0x20 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81028dc3>] intel_pmu_handle_irq+0x163/0x2e0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e6591>] perf_event_nmi_handler+0x21/0x30 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e5e6a>] do_nmi+0xfa/0x350 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e54b0>] nmi+0x20/0x30 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e2bef>] ? schedule+0x3f/0x60 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e4dd0>] ? _raw_spin_lock_irq+0x30/0x40 Apr 6 18:30:52 dellix [ 5358.187516] <<EOE>> Apr 6 18:30:52 [<ffffffff815e20d1>] __schedule+0xc1/0x8c0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e2bef>] schedule+0x3f/0x60 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81072e52>] do_exit+0x792/0x8a0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8107045a>] ? kmsg_dump+0x4a/0xe0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e5c3b>] oops_end+0xab/0xf0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815d990f>] no_context+0x214/0x223 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8101ad52>] ? nommu_map_page+0x42/0x80 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815d9ae7>] __bad_area_nosemaphore+0x1c9/0x1e8 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815d91bb>] ? pmd_offset+0x1f/0x25 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815d9b19>] bad_area_nosemaphore+0x13/0x15 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e8726>] do_page_fault+0x416/0x4f0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff812c3e4b>] ? vsnprintf+0x42b/0x5d0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff811944e4>] ? evict+0x104/0x1a0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e51f5>] page_fault+0x25/0x30 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81090841>] ? kthread_data+0x11/0x20 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8108c2a5>] ? wq_worker_sleeping+0x15/0xa0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e25f7>] __schedule+0x5e7/0x8c0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e2bef>] schedule+0x3f/0x60 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81072c6c>] do_exit+0x5ac/0x8a0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8107045a>] ? kmsg_dump+0x4a/0xe0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e5c3b>] oops_end+0xab/0xf0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81017868>] die+0x58/0x90 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815e5594>] do_trap+0xc4/0x170 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81014f05>] do_invalid_op+0x95/0xb0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e50968>] ? inode_go_sync+0x108/0x140 [gfs2] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e174ef>] ? __put_lkb+0x3f/0xd0 [dlm] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e1e55d>] ? dlm_put_lockspace+0x1d/0x30 [dlm] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815eeaab>] invalid_op+0x1b/0x20 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e50968>] ? inode_go_sync+0x108/0x140 [gfs2] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e4e9e9>] do_xmote+0xa9/0x250 [gfs2] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e4f1d8>] run_queue+0x148/0x2a0 [gfs2] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff81065d7e>] ? try_to_wake_up+0x22e/0x290 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e4f39d>] glock_work_func+0x6d/0x150 [gfs2] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffffa0e4f330>] ? run_queue+0x2a0/0x2a0 [gfs2] Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8108a7ad>] process_one_work+0x11d/0x470 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8108bbaf>] worker_thread+0x15f/0x350 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8108ba50>] ? manage_workers+0x230/0x230 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff8109056c>] kthread+0x8c/0xa0 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815eec34>] kernel_thread_helper+0x4/0x10 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff810904e0>] ? kthread_worker_fn+0x190/0x190 Apr 6 18:30:52 dellix [ 5358.187516] [<ffffffff815eec30>] ? gs_change+0x13/0x13 Apr 6 18:30:52 dellix [ 5358.187516] ---[ end trace a8655a0250dbfd42 ]--- Apr 6 18:31:09 dellix [ 5358.188010] ------------[ cut here ]------------ Apr 6 18:31:09 dellix [ 5358.188010] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0() Apr 6 18:31:09 dellix [ 5358.188010] Hardware name: Inspiron 530 Apr 6 18:31:09 dellix [ 5358.188010] Watchdog detected hard LOCKUP on cpu 0 Apr 6 18:31:09 dellix Apr 6 18:31:09 dellix [ 5358.188010] Modules linked in: Apr 6 18:31:09 binfmt_misc Apr 6 18:31:09 gfs2 Apr 6 18:31:09 dlm Apr 6 18:31:09 sha256_generic Apr 6 18:31:09 dm_crypt Apr 6 18:31:09 netconsole Apr 6 18:31:09 nvidia(P) Apr 6 18:31:09 ebtable_nat Apr 6 18:31:09 ebtables Apr 6 18:31:09 ipt_MASQUERADE Apr 6 18:31:09 iptable_nat Apr 6 18:31:09 nf_nat Apr 6 18:31:09 xt_CHECKSUM Apr 6 18:31:09 iptable_mangle Apr 6 18:31:09 tun Apr 6 18:31:09 bridge Apr 6 18:31:09 stp Apr 6 18:31:09 llc Apr 6 18:31:09 drbd Apr 6 18:31:09 lockd Apr 6 18:31:09 lru_cache Apr 6 18:31:09 ip6t_REJECT Apr 6 18:31:09 nf_conntrack_ipv6 Apr 6 18:31:09 nf_defrag_ipv6 Apr 6 18:31:09 nf_conntrack_ipv4 Apr 6 18:31:09 ip6table_filter Apr 6 18:31:09 nf_defrag_ipv4 Apr 6 18:31:09 ip6_tables Apr 6 18:31:09 xt_state Apr 6 18:31:09 nf_conntrack Apr 6 18:31:09 sunrpc Apr 6 18:31:09 usblp Apr 6 18:31:09 snd_hda_codec_realtek Apr 6 18:31:09 snd_hda_intel Apr 6 18:31:09 snd_hda_codec Apr 6 18:31:09 snd_hwdep Apr 6 18:31:09 snd_seq Apr 6 18:31:09 snd_seq_device Apr 6 18:31:09 snd_pcm Apr 6 18:31:09 i2c_i801 Apr 6 18:31:09 snd_timer Apr 6 18:31:09 snd Apr 6 18:31:09 soundcore Apr 6 18:31:09 snd_page_alloc Apr 6 18:31:09 iTCO_wdt Apr 6 18:31:09 i2c_core Apr 6 18:31:09 serio_raw Apr 6 18:31:09 iTCO_vendor_support Apr 6 18:31:09 e1000e Apr 6 18:31:09 microcode Apr 6 18:31:09 dcdbas Apr 6 18:31:09 uinput Apr 6 18:31:09 usb_storage Apr 6 18:31:09 [last unloaded: netconsole] Apr 6 18:31:09 dellix Apr 6 18:31:09 dellix [ 5358.188010] Pid: 14512, comm: rsyslogd Tainted: P D W O 3.2.10-6969.fc16.x86_64 #1 Apr 6 18:31:09 dellix [ 5358.188010] Call Trace: Apr 6 18:31:09 dellix [ 5358.188010] <NMI> Apr 6 18:31:09 [<ffffffff8106e72f>] warn_slowpath_common+0x7f/0xc0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8106e826>] warn_slowpath_fmt+0x46/0x50 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff810ddfba>] watchdog_overflow_callback+0x9a/0xc0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff811144e6>] __perf_event_overflow+0x96/0x1e0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81111ec1>] ? perf_event_update_userpage+0x11/0xc0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8102486a>] ? x86_perf_event_set_period+0xda/0x150 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81114e94>] perf_event_overflow+0x14/0x20 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81028dc3>] intel_pmu_handle_irq+0x163/0x2e0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e6591>] perf_event_nmi_handler+0x21/0x30 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e5e6a>] do_nmi+0xfa/0x350 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e54b0>] nmi+0x20/0x30 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e4d95>] ? _raw_spin_lock+0x25/0x30 Apr 6 18:31:09 dellix [ 5358.188010] <<EOE>> Apr 6 18:31:09 [<ffffffff81057325>] double_rq_lock+0x55/0x60 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81064dd1>] load_balance+0x251/0x850 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8101b9c9>] ? sched_clock+0x9/0x10 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81096ea8>] ? sched_clock_cpu+0xa8/0x110 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81056f53>] ? update_rq_clock+0x43/0x1a0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8105c98e>] ? update_shares+0xce/0x100 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e2875>] __schedule+0x865/0x8c0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8111a310>] ? sleep_on_page+0x20/0x20 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e2bef>] schedule+0x3f/0x60 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e2c9f>] io_schedule+0x8f/0xd0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8111a31e>] sleep_on_page_killable+0xe/0x40 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e34bf>] __wait_on_bit+0x5f/0x90 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e3461>] ? __wait_on_bit+0x1/0x90 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8111cbe0>] wait_on_page_bit_killable+0x80/0x90 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81090e60>] ? autoremove_wake_function+0x50/0x50 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8111cc96>] __lock_page_or_retry+0xa6/0xd0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8111d0b1>] filemap_fault+0x3f1/0x4a0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8113cecf>] __do_fault+0x6f/0x520 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff8113fec7>] handle_pte_fault+0xf7/0xac0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e23e4>] ? __schedule+0x3d4/0x8c0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81140c48>] handle_mm_fault+0x1f8/0x350 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e8452>] do_page_fault+0x142/0x4f0 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff81185d52>] ? path_put+0x22/0x30 Apr 6 18:31:09 dellix [ 5358.188010] [<ffffffff815e51f5>] page_fault+0x25/0x30 Apr 6 18:31:09 dellix [ 5358.188010] ---[ end trace a8655a0250dbfd43 ]---
It is tricky to tell, but I suspect that this bug may have already been fixed. It is possible that you've run into the bug which this patch fixes: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=35e478f42271673f79066a1ed008c6604621c6fe I'm not familiar with unison, so I've no idea what the i/o pattern is likely to be. You may, depending on the access patterns, and the nodes on which you are running the various commands, be running into contention on one or more inodes. That would explain the poor performance and also increase the likelihood of hitting the bug I mentioned above.
I can see it happening still in the exact some place after having rebuilt the module with the previous patch included. Further to that, I can also trigger the issue with a gfs2 module running on kernel 3.3 built out of the latest changes in HEAD bfecc60d8f6715ec6b38aa29c4f5a3570415dae0 unison by itself doesn't seem to be the problem as I have managed to run it from begin to end without incidents. But as soon as I turn on Jenkins, which every so often polls some git repos that exist in the fs the issue surfaces and the machines is brought down to a halt with a subsequent increase in noise triggered by what must be increases CPU activity. I can't yet triggered at will, e.g. not yet sure what is it exactly that makes it happen. It happens after a random amount of time after unison+jenkins are running. I'm not familiar with the codebase but would appreciate any hints on extra debugging that could lead to a resolution or more clues...
Hmm, lets go back to first principles then, I guess. I'm not familiar with Jenkins so I'm not at all sure what it is doing. How many nodes do you have? How many of those are running any kind of load? What kind of load is this? You mentioned syncing the content of the gfs2 filesystem with a usb disk, but which end is which? (i.e. is the gfs2 end all read i/o or is it the target of this operation rather than the source, or are there things going in both directions?) Are you using the noatime mount parameter? and if not could you add this since it is likely to reduce the load a fair bit if most of the i/o on the gfs2 filesystem is read only. I have been running the postmark and fsx benchmarks all yesterday on the latest -nmw git tree and I've not seen anything like this at all, so I wonder if this Jenkins thing is doing something odd. Are you using any non-default options, either on mount, or chattr +j or something like that? The part of the code that has hit the problem is in the glock code and specifically that part which writes back dirty inode data to disk before the inode is released. This is a generally well tested code path since it runs very often during any normal filesystem load, and especially at umount time and when there is memory pressure or remote nodes requesting glocks. Invalid opcode though is a very odd message to get, since the code should be read only, nothing should have been able to change it. I'd normally expect to see "NULL pointer dereference" or something like that, so I'm not really sure what to make of this one at the moment.
You're absolutely right... Context: This is a two node setup running GFS2 on top of DRBD. Currently, only one node has the FS mounted. The other node is just there as a "backup". Software: unison is a GUI based bi-directional rsync. In a nutshell it takes two endpoints and checks for differences. When it is finished you have the choice of manually syncing the differences to one side or to the other. The way I'm running it, and up to the point of the failure it is only doing reads as there is nothing to sync/write yet. Jenkins is the former Hudson it's just a CI tool. What it does in this context is poll regularly for changes some paths in the GFS volume which have git repositories. Addendum: No writes are involved in this apart from potentially atime updates ?! Triggering it: Triggering seems to be accomplished by having something like unison imposing a bit of load whilst running some git queries against the repos that exist on GFS. It's not yet conclusive... More info: This is how far I got on trying to get more info about what is going on. --- rpmbuild/BUILD/kernel-3.3.fc16/vanilla-3.3/fs/gfs2/glops.c 2012-03-18 23:15:34.000000000 +0000 +++ rpmbuild/BUILD/kernel-3.3.fc16/linux-3.3.x86_64/fs/gfs2/glops.c 2012-04-15 18:06:10.455113558 +0100 @@ -197,6 +197,19 @@ if (!test_and_clear_bit(GLF_DIRTY, &gl->gl_flags)) return; + if (gl->gl_state != LM_ST_EXCLUSIVE) { + + printk(KERN_CRIT "PEC DBG: BUG is ON"); + // struct inode *inode = ip->i_inode; + if(ip) { + printk("PEC DBG ino %lu mode 0%6.6o lk %d uid %d gid %d" + " sz %lu blks %lu cnt %u\n", + ip->i_inode.i_ino, ip->i_inode.i_mode, ip->i_inode.i_nlink, ip->i_inode.i_uid, + ip->i_inode.i_gid, ip->i_inode.i_size, ip->i_inode.i_blocks, + atomic_read(&ip->i_inode.i_count)); + } + printk(KERN_CRIT "PEC DBG: end of printing inode"); + } BUG_ON(gl->gl_state != LM_ST_EXCLUSIVE); ... and based on this snippet this what I've collected so far.... Apr 15 18:28:50 PEC DBG ino 66329 mode 0100444 lk 1 uid 1000 gid 1000 sz 9577 blks 32 cnt 0 Apr 16 22:05:49 PEC DBG ino 238990 mode 0100444 lk 1 uid 1000 gid 1000 sz 185 blks 8 cnt 1 Apr 17 20:26:23 PEC DBG ino 786388 mode 0100444 lk 1 uid 1000 gid 1000 sz 5160 blks 24 cnt 1 Apr 17 22:13:58 PEC DBG ino 238990 mode 0100444 lk 1 uid 1000 gid 1000 sz 185 blks 8 cnt 0 Apr 17 22:40:51 PEC DBG ino 262170 mode 0100444 lk 1 uid 1000 gid 1000 sz 20420 blks 48 cnt 0 Next: I'll try mounting with the noatime and try to trigger it.
Note that you shouldn't have to hack printk's into the source to see what is going on. There is a complete set of tracepoints for gfs2 which should allow you to see all the activity with respect to glocks, if required. Knowing whether noatime makes a difference will be a very useful data point I think.
noatime does make a difference. We haven't crashed yet. I've managed to emulate the behaviour of Jenkins and I've been submitting the server to 5 times more load since last night and it's still kicking.
# Mass update to all open bugs. Kernel 3.6.2-1.fc16 has just been pushed to updates. This update is a significant rebase from the previous version. Please retest with this kernel, and let us know if your problem has been fixed. In the event that you have upgraded to a newer release and the bug you reported is still present, please change the version field to the newest release you have encountered the issue with. Before doing so, please ensure you are testing the latest kernel update in that release and attach any new and relevant information you may have gathered. If you are not the original bug reporter and you still experience this bug, please file a new report, as it is possible that you may be seeing a different problem. (Please don't clone this bug, a fresh bug referencing this bug in the comment is sufficient).
Bug is still present with the most recent kernel... The only way to prevent it from happening is to mount the FS with the 'noatime' option. Version-Release number of selected component (if applicable): gfs2-cluster-3.1.3-1.fc16.x86_64 gfs2-utils-3.1.3-1.fc16.x86_64 kernel-3.6.2-1.fc16.x86_64 netconsole dump: Oct 25 11:23:52 dellix [ 846.769711] ------------[ cut here ]------------ Oct 25 11:23:52 dellix [ 846.770003] kernel BUG at fs/gfs2/glops.c:199! Oct 25 11:23:52 dellix [ 846.770003] invalid opcode: 0000 [#1] Oct 25 11:23:52 dellix SMP Oct 25 11:23:52 dellix Oct 25 11:23:52 dellix [ 846.770003] Modules linked in: Oct 25 11:23:52 gfs2 Oct 25 11:23:52 sha256_generic Oct 25 11:23:52 dm_crypt Oct 25 11:23:52 dlm Oct 25 11:23:52 sctp Oct 25 11:23:52 libcrc32c Oct 25 11:23:52 netconsole Oct 25 11:23:52 ebtable_nat Oct 25 11:23:52 ebtables Oct 25 11:23:52 ipt_MASQUERADE Oct 25 11:23:52 iptable_nat Oct 25 11:23:52 nf_nat Oct 25 11:23:52 xt_CHECKSUM Oct 25 11:23:52 iptable_mangle Oct 25 11:23:52 tun Oct 25 11:23:52 bridge Oct 25 11:23:52 stp Oct 25 11:23:52 llc Oct 25 11:23:52 drbd Oct 25 11:23:52 lru_cache Oct 25 11:23:52 ip6t_REJECT Oct 25 11:23:52 nf_conntrack_ipv4 Oct 25 11:23:52 nf_conntrack_ipv6 Oct 25 11:23:52 nf_defrag_ipv4 Oct 25 11:23:52 nf_defrag_ipv6 Oct 25 11:23:52 xt_state Oct 25 11:23:52 nf_conntrack Oct 25 11:23:52 ip6table_filter Oct 25 11:23:52 ip6_tables Oct 25 11:23:52 hwmon_vid Oct 25 11:23:52 binfmt_misc Oct 25 11:23:52 snd_hda_codec_realtek Oct 25 11:23:52 snd_hda_intel Oct 25 11:23:52 snd_hda_codec Oct 25 11:23:52 snd_hwdep Oct 25 11:23:52 iTCO_wdt Oct 25 11:23:52 snd_seq Oct 25 11:23:52 snd_seq_device Oct 25 11:23:52 snd_pcm Oct 25 11:23:52 iTCO_vendor_support Oct 25 11:23:52 snd_timer Oct 25 11:23:52 snd Oct 25 11:23:52 lpc_ich Oct 25 11:23:52 serio_raw Oct 25 11:23:52 coretemp Oct 25 11:23:52 microcode Oct 25 11:23:52 soundcore Oct 25 11:23:52 i2c_i801 Oct 25 11:23:52 snd_page_alloc Oct 25 11:23:52 dcdbas Oct 25 11:23:52 e1000e Oct 25 11:23:52 mfd_core Oct 25 11:23:52 usb_storage Oct 25 11:23:52 nouveau Oct 25 11:23:52 ttm Oct 25 11:23:52 drm_kms_helper Oct 25 11:23:52 drm Oct 25 11:23:53 i2c_algo_bit Oct 25 11:23:53 i2c_core Oct 25 11:23:53 mxm_wmi Oct 25 11:23:53 video Oct 25 11:23:53 wmi Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] CPU 1 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] Pid: 7299, comm: kworker/1:8H Not tainted 3.6.2-1.fc16.x86_64 #1 Dell Inc. Inspiron 530/0RY007 Oct 25 11:23:53 dellix [ 846.770003] RIP: 0010:[<ffffffffa0416cc8>] Oct 25 11:23:53 [<ffffffffa0416cc8>] inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] RSP: 0000:ffff88003cf43c90 EFLAGS: 00010202 Oct 25 11:23:53 dellix [ 846.770003] RAX: 0000000000000003 RBX: ffff88003a7a6970 RCX: 0000000000000003 Oct 25 11:23:53 dellix [ 846.770003] RDX: ffffffffa0436b80 RSI: 0000000000000000 RDI: ffff88003a7a6970 Oct 25 11:23:53 dellix [ 846.770003] RBP: ffff88003cf43cb0 R08: ffff88003a7a6a98 R09: ffff8800c90d8000 Oct 25 11:23:53 dellix [ 846.770003] R10: 000000000003907f R11: 0000000000000000 R12: ffff88003b8de760 Oct 25 11:23:53 dellix [ 846.770003] R13: ffff88003a7a6b30 R14: ffff88003a7a69c0 R15: ffffffffa04155c0 Oct 25 11:23:53 dellix [ 846.770003] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000 Oct 25 11:23:53 dellix [ 846.770003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Oct 25 11:23:53 dellix [ 846.770003] CR2: 00007f796830f040 CR3: 0000000076f77000 CR4: 00000000000007e0 Oct 25 11:23:53 dellix [ 846.770003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 25 11:23:53 dellix [ 846.770003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 25 11:23:53 dellix [ 846.770003] Process kworker/1:8H (pid: 7299, threadinfo ffff88003cf42000, task ffff8800274fdc80) Oct 25 11:23:53 dellix [ 846.770003] Stack: Oct 25 11:23:53 dellix [ 846.770003] ffff88003cf1e270 Oct 25 11:23:53 ffff88003a7a6970 Oct 25 11:23:53 ffff88003a7a69c0 Oct 25 11:23:53 ffff88003a7a69c0 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] ffff88003cf43d20 Oct 25 11:23:53 ffffffffa0414c76 Oct 25 11:23:53 ffff8800274fdcc8 Oct 25 11:23:53 00ff8800cfc93db0 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] ffff88003bd49000 Oct 25 11:23:53 ffffffffa0436b80 Oct 25 11:23:53 ffff88003a7a69c0 Oct 25 11:23:53 0000000000000000 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] Call Trace: Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa0414c76>] do_xmote+0xc6/0x270 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa0415468>] run_queue+0x148/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa041460e>] ? gfs2_glock_put+0x4e/0x190 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa041562d>] glock_work_func+0x6d/0x150 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81079c7a>] process_one_work+0x13a/0x530 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff8107b07e>] worker_thread+0x18e/0x510 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff8107aef0>] ? manage_workers+0x310/0x310 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81080153>] kthread+0x93/0xa0 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81622244>] kernel_thread_helper+0x4/0x10 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff810800c0>] ? flush_kthread_worker+0xb0/0xb0 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81622240>] ? gs_change+0x13/0x13 Oct 25 11:23:53 dellix [ 846.770003] Code: Oct 25 11:23:53 dellix 85 Oct 25 11:23:53 dellix c0 Oct 25 11:23:53 dellix 0f Oct 25 11:23:53 dellix 84 Oct 25 11:23:53 dellix 71 Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix 49 Oct 25 11:23:53 dellix 8b Oct 25 11:23:53 dellix 7c Oct 25 11:23:53 dellix 24 Oct 25 11:23:53 dellix 30 Oct 25 11:23:53 dellix 31 Oct 25 11:23:53 dellix c9 Oct 25 11:23:53 dellix 31 Oct 25 11:23:53 dellix d2 Oct 25 11:23:53 dellix 31 Oct 25 11:23:53 dellix f6 Oct 25 11:23:53 dellix e8 Oct 25 11:23:53 dellix 4b Oct 25 11:23:53 dellix c2 Oct 25 11:23:53 dellix d3 Oct 25 11:23:53 dellix e0 Oct 25 11:23:53 dellix e9 Oct 25 11:23:53 dellix 5c Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix 83 Oct 25 11:23:53 dellix f8 Oct 25 11:23:53 dellix e4 Oct 25 11:23:53 dellix 74 Oct 25 11:23:53 dellix 23 Oct 25 11:23:53 dellix f0 Oct 25 11:23:53 dellix 41 Oct 25 11:23:53 dellix 80 Oct 25 11:23:53 dellix 4c Oct 25 11:23:53 dellix 24 Oct 25 11:23:53 dellix 7b Oct 25 11:23:53 dellix 02 Oct 25 11:23:53 dellix eb Oct 25 11:23:53 dellix 9b Oct 25 11:23:53 dellix f> Oct 25 11:23:53 dellix 0b Oct 25 11:23:53 dellix 83 Oct 25 11:23:53 dellix f8 Oct 25 11:23:53 dellix e4 Oct 25 11:23:53 dellix 0f Oct 25 11:23:53 dellix 1f Oct 25 11:23:53 dellix 00 Oct 25 11:23:53 dellix 74 Oct 25 11:23:53 dellix 1d Oct 25 11:23:53 dellix f0 Oct 25 11:23:53 dellix 41 Oct 25 11:23:53 dellix 80 Oct 25 11:23:53 dellix 4d Oct 25 11:23:53 dellix 7b Oct 25 11:23:53 dellix 02 Oct 25 11:23:53 dellix 0f Oct 25 11:23:53 dellix 1f Oct 25 11:23:53 dellix 84 Oct 25 11:23:53 dellix 00 Oct 25 11:23:53 dellix 00 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.795207] RIP Oct 25 11:23:53 [<ffffffffa0416cc8>] inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.795207] RSP <ffff88003cf43c90> Oct 25 11:23:53 dellix [ 846.822877] ---[ end trace 279603713a1c265f ]--- Oct 25 11:23:53 dellix [ 846.823305] BUG: unable to handle kernel Oct 25 11:23:53 paging request Oct 25 11:23:53 at fffffffffffffff8 Oct 25 11:23:53 dellix [ 846.823705] IP: Oct 25 11:23:53 [<ffffffff81080421>] kthread_data+0x11/0x20 Oct 25 11:23:53 dellix [ 846.824001] PGD 1c0d067 Oct 25 11:23:53 PUD 1c0e067 Oct 25 11:23:53 PMD 0 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Oops: 0000 [#2] Oct 25 11:23:53 dellix SMP Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Modules linked in: Oct 25 11:23:53 gfs2 Oct 25 11:23:53 sha256_generic Oct 25 11:23:53 dm_crypt Oct 25 11:23:53 dlm Oct 25 11:23:53 sctp Oct 25 11:23:53 libcrc32c Oct 25 11:23:53 netconsole Oct 25 11:23:53 ebtable_nat Oct 25 11:23:53 ebtables Oct 25 11:23:53 ipt_MASQUERADE Oct 25 11:23:53 iptable_nat Oct 25 11:23:53 nf_nat Oct 25 11:23:53 xt_CHECKSUM Oct 25 11:23:53 iptable_mangle Oct 25 11:23:53 tun Oct 25 11:23:53 bridge Oct 25 11:23:53 stp Oct 25 11:23:53 llc Oct 25 11:23:53 drbd Oct 25 11:23:53 lru_cache Oct 25 11:23:53 ip6t_REJECT Oct 25 11:23:53 nf_conntrack_ipv4 Oct 25 11:23:53 nf_conntrack_ipv6 Oct 25 11:23:53 nf_defrag_ipv4 Oct 25 11:23:53 nf_defrag_ipv6 Oct 25 11:23:53 xt_state Oct 25 11:23:53 nf_conntrack Oct 25 11:23:53 ip6table_filter Oct 25 11:23:53 ip6_tables Oct 25 11:23:53 hwmon_vid Oct 25 11:23:53 binfmt_misc Oct 25 11:23:53 snd_hda_codec_realtek Oct 25 11:23:53 snd_hda_intel Oct 25 11:23:53 snd_hda_codec Oct 25 11:23:53 snd_hwdep Oct 25 11:23:53 iTCO_wdt Oct 25 11:23:53 snd_seq Oct 25 11:23:53 snd_seq_device Oct 25 11:23:53 snd_pcm Oct 25 11:23:53 iTCO_vendor_support Oct 25 11:23:53 snd_timer Oct 25 11:23:53 snd Oct 25 11:23:53 lpc_ich Oct 25 11:23:53 serio_raw Oct 25 11:23:53 coretemp Oct 25 11:23:53 microcode Oct 25 11:23:53 soundcore Oct 25 11:23:53 i2c_i801 Oct 25 11:23:53 snd_page_alloc Oct 25 11:23:53 dcdbas Oct 25 11:23:53 e1000e Oct 25 11:23:53 mfd_core Oct 25 11:23:53 usb_storage Oct 25 11:23:53 nouveau Oct 25 11:23:53 ttm Oct 25 11:23:53 drm_kms_helper Oct 25 11:23:53 drm Oct 25 11:23:53 i2c_algo_bit Oct 25 11:23:53 i2c_core Oct 25 11:23:53 mxm_wmi Oct 25 11:23:53 video Oct 25 11:23:53 wmi Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] CPU 1 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Pid: 7299, comm: kworker/1:8H Tainted: G D 3.6.2-1.fc16.x86_64 #1 Dell Inc. Inspiron 530/0RY007 Oct 25 11:23:53 dellix [ 846.824001] RIP: 0010:[<ffffffff81080421>] Oct 25 11:23:53 [<ffffffff81080421>] kthread_data+0x11/0x20 Oct 25 11:23:53 dellix [ 846.824001] RSP: 0018:ffff88003cf43940 EFLAGS: 00010096 Oct 25 11:23:53 dellix [ 846.824001] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001 Oct 25 11:23:53 dellix [ 846.824001] RDX: ffff8800cbc05000 RSI: 0000000000000001 RDI: ffff8800274fdc80 Oct 25 11:23:53 dellix [ 846.824001] RBP: ffff88003cf43958 R08: ffff8800274fdcf0 R09: 0000000000000001 Oct 25 11:23:53 dellix [ 846.824001] R10: 0000000000000000 R11: 000000000000002f R12: 0000000000000001 Oct 25 11:23:53 dellix [ 846.824001] R13: ffff8800274fe050 R14: ffff88003cf43748 R15: ffff8800274fdf58 Oct 25 11:23:53 dellix [ 846.824001] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000 Oct 25 11:23:53 dellix [ 846.824001] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Oct 25 11:23:53 dellix [ 846.824001] CR2: fffffffffffffff8 CR3: 0000000069b83000 CR4: 00000000000007e0 Oct 25 11:23:53 dellix [ 846.824001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 25 11:23:53 dellix [ 846.824001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 25 11:23:53 dellix [ 846.824001] Process kworker/1:8H (pid: 7299, threadinfo ffff88003cf42000, task ffff8800274fdc80) Oct 25 11:23:53 dellix [ 846.824001] Stack: Oct 25 11:23:53 dellix [ 846.824001] ffffffff8107b9a5 Oct 25 11:23:53 ffff88003cf43958 Oct 25 11:23:53 ffff8800cfc93d40 Oct 25 11:23:53 ffff88003cf439d8 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] ffffffff81617b2f Oct 25 11:23:53 ffff88003cf43998 Oct 25 11:23:53 ffffffff812baf42 Oct 25 11:23:53 ffff8800274fdc80 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] ffff88003cf43fd8 Oct 25 11:23:53 ffff88003cf43fd8 Oct 25 11:23:53 ffff88003cf43fd8 Oct 25 11:23:53 ffff88003cf439c8 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Call Trace: Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8107b9a5>] ? wq_worker_sleeping+0x15/0xc0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81617b2f>] __schedule+0x5df/0x7c0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff812baf42>] ? put_io_context+0x72/0xa0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81618019>] schedule+0x29/0x70 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff810628cd>] do_exit+0x58d/0x8d0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8160e90a>] ? printk+0x61/0x63 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8161a23c>] oops_end+0xac/0xf0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff810177f8>] die+0x58/0x90 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81619b14>] do_trap+0xc4/0x170 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81014e85>] do_invalid_op+0x95/0xb0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0416cc8>] ? inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0432210>] ? gfs2_control_func+0x3e0/0x3e0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff811385b2>] ? pagevec_lookup+0x22/0x30 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04317f0>] ? gdlm_lock+0x330/0x330 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0432210>] ? gfs2_control_func+0x3e0/0x3e0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff816220bb>] invalid_op+0x1b/0x20 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0416cc8>] ? inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0414c76>] do_xmote+0xc6/0x270 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0415468>] run_queue+0x148/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa041460e>] ? gfs2_glock_put+0x4e/0x190 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa041562d>] glock_work_func+0x6d/0x150 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81079c7a>] process_one_work+0x13a/0x530 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8107b07e>] worker_thread+0x18e/0x510 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8107aef0>] ? manage_workers+0x310/0x310 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81080153>] kthread+0x93/0xa0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81622244>] kernel_thread_helper+0x4/0x10 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff810800c0>] ? flush_kthread_worker+0xb0/0xb0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81622240>] ? gs_change+0x13/0x13 Oct 25 11:23:53 dellix [ 846.824001] Code: Oct 25 11:23:53 dellix 81 Oct 25 11:23:53 dellix e8 Oct 25 11:23:53 dellix e4 Oct 25 11:23:53 dellix c2 Oct 25 11:23:53 dellix fd Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix e9
Fix posted to 3.7: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=3d1626889a64bd5a661544d582036a0a02104a60
Fix also posted to the 3.6 stable tree.
Fix has been added to the 3.6 stable tree, so it should appear in the next stable tree release.
Now in the 3.6.8 stable tree