Bug 810713
Summary: | Complete system hang triggered by GFS2 bug... | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Paulo Castro <pauloedgarcastro> |
Component: | kernel | Assignee: | Ben Marzinski <bmarzins> |
Status: | CLOSED UPSTREAM | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | 16 | CC: | adas, anprice, gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda, mhilgart, rpeterso, swhiteho |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2012-11-27 10:01:24 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Paulo Castro
2012-04-08 15:04:46 UTC
It is tricky to tell, but I suspect that this bug may have already been fixed. It is possible that you've run into the bug which this patch fixes: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=35e478f42271673f79066a1ed008c6604621c6fe I'm not familiar with unison, so I've no idea what the i/o pattern is likely to be. You may, depending on the access patterns, and the nodes on which you are running the various commands, be running into contention on one or more inodes. That would explain the poor performance and also increase the likelihood of hitting the bug I mentioned above. I can see it happening still in the exact some place after having rebuilt the module with the previous patch included. Further to that, I can also trigger the issue with a gfs2 module running on kernel 3.3 built out of the latest changes in HEAD bfecc60d8f6715ec6b38aa29c4f5a3570415dae0 unison by itself doesn't seem to be the problem as I have managed to run it from begin to end without incidents. But as soon as I turn on Jenkins, which every so often polls some git repos that exist in the fs the issue surfaces and the machines is brought down to a halt with a subsequent increase in noise triggered by what must be increases CPU activity. I can't yet triggered at will, e.g. not yet sure what is it exactly that makes it happen. It happens after a random amount of time after unison+jenkins are running. I'm not familiar with the codebase but would appreciate any hints on extra debugging that could lead to a resolution or more clues... Hmm, lets go back to first principles then, I guess. I'm not familiar with Jenkins so I'm not at all sure what it is doing. How many nodes do you have? How many of those are running any kind of load? What kind of load is this? You mentioned syncing the content of the gfs2 filesystem with a usb disk, but which end is which? (i.e. is the gfs2 end all read i/o or is it the target of this operation rather than the source, or are there things going in both directions?) Are you using the noatime mount parameter? and if not could you add this since it is likely to reduce the load a fair bit if most of the i/o on the gfs2 filesystem is read only. I have been running the postmark and fsx benchmarks all yesterday on the latest -nmw git tree and I've not seen anything like this at all, so I wonder if this Jenkins thing is doing something odd. Are you using any non-default options, either on mount, or chattr +j or something like that? The part of the code that has hit the problem is in the glock code and specifically that part which writes back dirty inode data to disk before the inode is released. This is a generally well tested code path since it runs very often during any normal filesystem load, and especially at umount time and when there is memory pressure or remote nodes requesting glocks. Invalid opcode though is a very odd message to get, since the code should be read only, nothing should have been able to change it. I'd normally expect to see "NULL pointer dereference" or something like that, so I'm not really sure what to make of this one at the moment. You're absolutely right... Context: This is a two node setup running GFS2 on top of DRBD. Currently, only one node has the FS mounted. The other node is just there as a "backup". Software: unison is a GUI based bi-directional rsync. In a nutshell it takes two endpoints and checks for differences. When it is finished you have the choice of manually syncing the differences to one side or to the other. The way I'm running it, and up to the point of the failure it is only doing reads as there is nothing to sync/write yet. Jenkins is the former Hudson it's just a CI tool. What it does in this context is poll regularly for changes some paths in the GFS volume which have git repositories. Addendum: No writes are involved in this apart from potentially atime updates ?! Triggering it: Triggering seems to be accomplished by having something like unison imposing a bit of load whilst running some git queries against the repos that exist on GFS. It's not yet conclusive... More info: This is how far I got on trying to get more info about what is going on. --- rpmbuild/BUILD/kernel-3.3.fc16/vanilla-3.3/fs/gfs2/glops.c 2012-03-18 23:15:34.000000000 +0000 +++ rpmbuild/BUILD/kernel-3.3.fc16/linux-3.3.x86_64/fs/gfs2/glops.c 2012-04-15 18:06:10.455113558 +0100 @@ -197,6 +197,19 @@ if (!test_and_clear_bit(GLF_DIRTY, &gl->gl_flags)) return; + if (gl->gl_state != LM_ST_EXCLUSIVE) { + + printk(KERN_CRIT "PEC DBG: BUG is ON"); + // struct inode *inode = ip->i_inode; + if(ip) { + printk("PEC DBG ino %lu mode 0%6.6o lk %d uid %d gid %d" + " sz %lu blks %lu cnt %u\n", + ip->i_inode.i_ino, ip->i_inode.i_mode, ip->i_inode.i_nlink, ip->i_inode.i_uid, + ip->i_inode.i_gid, ip->i_inode.i_size, ip->i_inode.i_blocks, + atomic_read(&ip->i_inode.i_count)); + } + printk(KERN_CRIT "PEC DBG: end of printing inode"); + } BUG_ON(gl->gl_state != LM_ST_EXCLUSIVE); ... and based on this snippet this what I've collected so far.... Apr 15 18:28:50 PEC DBG ino 66329 mode 0100444 lk 1 uid 1000 gid 1000 sz 9577 blks 32 cnt 0 Apr 16 22:05:49 PEC DBG ino 238990 mode 0100444 lk 1 uid 1000 gid 1000 sz 185 blks 8 cnt 1 Apr 17 20:26:23 PEC DBG ino 786388 mode 0100444 lk 1 uid 1000 gid 1000 sz 5160 blks 24 cnt 1 Apr 17 22:13:58 PEC DBG ino 238990 mode 0100444 lk 1 uid 1000 gid 1000 sz 185 blks 8 cnt 0 Apr 17 22:40:51 PEC DBG ino 262170 mode 0100444 lk 1 uid 1000 gid 1000 sz 20420 blks 48 cnt 0 Next: I'll try mounting with the noatime and try to trigger it. Note that you shouldn't have to hack printk's into the source to see what is going on. There is a complete set of tracepoints for gfs2 which should allow you to see all the activity with respect to glocks, if required. Knowing whether noatime makes a difference will be a very useful data point I think. noatime does make a difference. We haven't crashed yet. I've managed to emulate the behaviour of Jenkins and I've been submitting the server to 5 times more load since last night and it's still kicking. # Mass update to all open bugs. Kernel 3.6.2-1.fc16 has just been pushed to updates. This update is a significant rebase from the previous version. Please retest with this kernel, and let us know if your problem has been fixed. In the event that you have upgraded to a newer release and the bug you reported is still present, please change the version field to the newest release you have encountered the issue with. Before doing so, please ensure you are testing the latest kernel update in that release and attach any new and relevant information you may have gathered. If you are not the original bug reporter and you still experience this bug, please file a new report, as it is possible that you may be seeing a different problem. (Please don't clone this bug, a fresh bug referencing this bug in the comment is sufficient). Bug is still present with the most recent kernel... The only way to prevent it from happening is to mount the FS with the 'noatime' option. Version-Release number of selected component (if applicable): gfs2-cluster-3.1.3-1.fc16.x86_64 gfs2-utils-3.1.3-1.fc16.x86_64 kernel-3.6.2-1.fc16.x86_64 netconsole dump: Oct 25 11:23:52 dellix [ 846.769711] ------------[ cut here ]------------ Oct 25 11:23:52 dellix [ 846.770003] kernel BUG at fs/gfs2/glops.c:199! Oct 25 11:23:52 dellix [ 846.770003] invalid opcode: 0000 [#1] Oct 25 11:23:52 dellix SMP Oct 25 11:23:52 dellix Oct 25 11:23:52 dellix [ 846.770003] Modules linked in: Oct 25 11:23:52 gfs2 Oct 25 11:23:52 sha256_generic Oct 25 11:23:52 dm_crypt Oct 25 11:23:52 dlm Oct 25 11:23:52 sctp Oct 25 11:23:52 libcrc32c Oct 25 11:23:52 netconsole Oct 25 11:23:52 ebtable_nat Oct 25 11:23:52 ebtables Oct 25 11:23:52 ipt_MASQUERADE Oct 25 11:23:52 iptable_nat Oct 25 11:23:52 nf_nat Oct 25 11:23:52 xt_CHECKSUM Oct 25 11:23:52 iptable_mangle Oct 25 11:23:52 tun Oct 25 11:23:52 bridge Oct 25 11:23:52 stp Oct 25 11:23:52 llc Oct 25 11:23:52 drbd Oct 25 11:23:52 lru_cache Oct 25 11:23:52 ip6t_REJECT Oct 25 11:23:52 nf_conntrack_ipv4 Oct 25 11:23:52 nf_conntrack_ipv6 Oct 25 11:23:52 nf_defrag_ipv4 Oct 25 11:23:52 nf_defrag_ipv6 Oct 25 11:23:52 xt_state Oct 25 11:23:52 nf_conntrack Oct 25 11:23:52 ip6table_filter Oct 25 11:23:52 ip6_tables Oct 25 11:23:52 hwmon_vid Oct 25 11:23:52 binfmt_misc Oct 25 11:23:52 snd_hda_codec_realtek Oct 25 11:23:52 snd_hda_intel Oct 25 11:23:52 snd_hda_codec Oct 25 11:23:52 snd_hwdep Oct 25 11:23:52 iTCO_wdt Oct 25 11:23:52 snd_seq Oct 25 11:23:52 snd_seq_device Oct 25 11:23:52 snd_pcm Oct 25 11:23:52 iTCO_vendor_support Oct 25 11:23:52 snd_timer Oct 25 11:23:52 snd Oct 25 11:23:52 lpc_ich Oct 25 11:23:52 serio_raw Oct 25 11:23:52 coretemp Oct 25 11:23:52 microcode Oct 25 11:23:52 soundcore Oct 25 11:23:52 i2c_i801 Oct 25 11:23:52 snd_page_alloc Oct 25 11:23:52 dcdbas Oct 25 11:23:52 e1000e Oct 25 11:23:52 mfd_core Oct 25 11:23:52 usb_storage Oct 25 11:23:52 nouveau Oct 25 11:23:52 ttm Oct 25 11:23:52 drm_kms_helper Oct 25 11:23:52 drm Oct 25 11:23:53 i2c_algo_bit Oct 25 11:23:53 i2c_core Oct 25 11:23:53 mxm_wmi Oct 25 11:23:53 video Oct 25 11:23:53 wmi Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] CPU 1 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] Pid: 7299, comm: kworker/1:8H Not tainted 3.6.2-1.fc16.x86_64 #1 Dell Inc. Inspiron 530/0RY007 Oct 25 11:23:53 dellix [ 846.770003] RIP: 0010:[<ffffffffa0416cc8>] Oct 25 11:23:53 [<ffffffffa0416cc8>] inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] RSP: 0000:ffff88003cf43c90 EFLAGS: 00010202 Oct 25 11:23:53 dellix [ 846.770003] RAX: 0000000000000003 RBX: ffff88003a7a6970 RCX: 0000000000000003 Oct 25 11:23:53 dellix [ 846.770003] RDX: ffffffffa0436b80 RSI: 0000000000000000 RDI: ffff88003a7a6970 Oct 25 11:23:53 dellix [ 846.770003] RBP: ffff88003cf43cb0 R08: ffff88003a7a6a98 R09: ffff8800c90d8000 Oct 25 11:23:53 dellix [ 846.770003] R10: 000000000003907f R11: 0000000000000000 R12: ffff88003b8de760 Oct 25 11:23:53 dellix [ 846.770003] R13: ffff88003a7a6b30 R14: ffff88003a7a69c0 R15: ffffffffa04155c0 Oct 25 11:23:53 dellix [ 846.770003] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000 Oct 25 11:23:53 dellix [ 846.770003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Oct 25 11:23:53 dellix [ 846.770003] CR2: 00007f796830f040 CR3: 0000000076f77000 CR4: 00000000000007e0 Oct 25 11:23:53 dellix [ 846.770003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 25 11:23:53 dellix [ 846.770003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 25 11:23:53 dellix [ 846.770003] Process kworker/1:8H (pid: 7299, threadinfo ffff88003cf42000, task ffff8800274fdc80) Oct 25 11:23:53 dellix [ 846.770003] Stack: Oct 25 11:23:53 dellix [ 846.770003] ffff88003cf1e270 Oct 25 11:23:53 ffff88003a7a6970 Oct 25 11:23:53 ffff88003a7a69c0 Oct 25 11:23:53 ffff88003a7a69c0 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] ffff88003cf43d20 Oct 25 11:23:53 ffffffffa0414c76 Oct 25 11:23:53 ffff8800274fdcc8 Oct 25 11:23:53 00ff8800cfc93db0 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] ffff88003bd49000 Oct 25 11:23:53 ffffffffa0436b80 Oct 25 11:23:53 ffff88003a7a69c0 Oct 25 11:23:53 0000000000000000 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.770003] Call Trace: Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa0414c76>] do_xmote+0xc6/0x270 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa0415468>] run_queue+0x148/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa041460e>] ? gfs2_glock_put+0x4e/0x190 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffffa041562d>] glock_work_func+0x6d/0x150 [gfs2] Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81079c7a>] process_one_work+0x13a/0x530 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff8107b07e>] worker_thread+0x18e/0x510 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff8107aef0>] ? manage_workers+0x310/0x310 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81080153>] kthread+0x93/0xa0 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81622244>] kernel_thread_helper+0x4/0x10 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff810800c0>] ? flush_kthread_worker+0xb0/0xb0 Oct 25 11:23:53 dellix [ 846.770003] [<ffffffff81622240>] ? gs_change+0x13/0x13 Oct 25 11:23:53 dellix [ 846.770003] Code: Oct 25 11:23:53 dellix 85 Oct 25 11:23:53 dellix c0 Oct 25 11:23:53 dellix 0f Oct 25 11:23:53 dellix 84 Oct 25 11:23:53 dellix 71 Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix 49 Oct 25 11:23:53 dellix 8b Oct 25 11:23:53 dellix 7c Oct 25 11:23:53 dellix 24 Oct 25 11:23:53 dellix 30 Oct 25 11:23:53 dellix 31 Oct 25 11:23:53 dellix c9 Oct 25 11:23:53 dellix 31 Oct 25 11:23:53 dellix d2 Oct 25 11:23:53 dellix 31 Oct 25 11:23:53 dellix f6 Oct 25 11:23:53 dellix e8 Oct 25 11:23:53 dellix 4b Oct 25 11:23:53 dellix c2 Oct 25 11:23:53 dellix d3 Oct 25 11:23:53 dellix e0 Oct 25 11:23:53 dellix e9 Oct 25 11:23:53 dellix 5c Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix 83 Oct 25 11:23:53 dellix f8 Oct 25 11:23:53 dellix e4 Oct 25 11:23:53 dellix 74 Oct 25 11:23:53 dellix 23 Oct 25 11:23:53 dellix f0 Oct 25 11:23:53 dellix 41 Oct 25 11:23:53 dellix 80 Oct 25 11:23:53 dellix 4c Oct 25 11:23:53 dellix 24 Oct 25 11:23:53 dellix 7b Oct 25 11:23:53 dellix 02 Oct 25 11:23:53 dellix eb Oct 25 11:23:53 dellix 9b Oct 25 11:23:53 dellix f> Oct 25 11:23:53 dellix 0b Oct 25 11:23:53 dellix 83 Oct 25 11:23:53 dellix f8 Oct 25 11:23:53 dellix e4 Oct 25 11:23:53 dellix 0f Oct 25 11:23:53 dellix 1f Oct 25 11:23:53 dellix 00 Oct 25 11:23:53 dellix 74 Oct 25 11:23:53 dellix 1d Oct 25 11:23:53 dellix f0 Oct 25 11:23:53 dellix 41 Oct 25 11:23:53 dellix 80 Oct 25 11:23:53 dellix 4d Oct 25 11:23:53 dellix 7b Oct 25 11:23:53 dellix 02 Oct 25 11:23:53 dellix 0f Oct 25 11:23:53 dellix 1f Oct 25 11:23:53 dellix 84 Oct 25 11:23:53 dellix 00 Oct 25 11:23:53 dellix 00 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.795207] RIP Oct 25 11:23:53 [<ffffffffa0416cc8>] inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.795207] RSP <ffff88003cf43c90> Oct 25 11:23:53 dellix [ 846.822877] ---[ end trace 279603713a1c265f ]--- Oct 25 11:23:53 dellix [ 846.823305] BUG: unable to handle kernel Oct 25 11:23:53 paging request Oct 25 11:23:53 at fffffffffffffff8 Oct 25 11:23:53 dellix [ 846.823705] IP: Oct 25 11:23:53 [<ffffffff81080421>] kthread_data+0x11/0x20 Oct 25 11:23:53 dellix [ 846.824001] PGD 1c0d067 Oct 25 11:23:53 PUD 1c0e067 Oct 25 11:23:53 PMD 0 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Oops: 0000 [#2] Oct 25 11:23:53 dellix SMP Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Modules linked in: Oct 25 11:23:53 gfs2 Oct 25 11:23:53 sha256_generic Oct 25 11:23:53 dm_crypt Oct 25 11:23:53 dlm Oct 25 11:23:53 sctp Oct 25 11:23:53 libcrc32c Oct 25 11:23:53 netconsole Oct 25 11:23:53 ebtable_nat Oct 25 11:23:53 ebtables Oct 25 11:23:53 ipt_MASQUERADE Oct 25 11:23:53 iptable_nat Oct 25 11:23:53 nf_nat Oct 25 11:23:53 xt_CHECKSUM Oct 25 11:23:53 iptable_mangle Oct 25 11:23:53 tun Oct 25 11:23:53 bridge Oct 25 11:23:53 stp Oct 25 11:23:53 llc Oct 25 11:23:53 drbd Oct 25 11:23:53 lru_cache Oct 25 11:23:53 ip6t_REJECT Oct 25 11:23:53 nf_conntrack_ipv4 Oct 25 11:23:53 nf_conntrack_ipv6 Oct 25 11:23:53 nf_defrag_ipv4 Oct 25 11:23:53 nf_defrag_ipv6 Oct 25 11:23:53 xt_state Oct 25 11:23:53 nf_conntrack Oct 25 11:23:53 ip6table_filter Oct 25 11:23:53 ip6_tables Oct 25 11:23:53 hwmon_vid Oct 25 11:23:53 binfmt_misc Oct 25 11:23:53 snd_hda_codec_realtek Oct 25 11:23:53 snd_hda_intel Oct 25 11:23:53 snd_hda_codec Oct 25 11:23:53 snd_hwdep Oct 25 11:23:53 iTCO_wdt Oct 25 11:23:53 snd_seq Oct 25 11:23:53 snd_seq_device Oct 25 11:23:53 snd_pcm Oct 25 11:23:53 iTCO_vendor_support Oct 25 11:23:53 snd_timer Oct 25 11:23:53 snd Oct 25 11:23:53 lpc_ich Oct 25 11:23:53 serio_raw Oct 25 11:23:53 coretemp Oct 25 11:23:53 microcode Oct 25 11:23:53 soundcore Oct 25 11:23:53 i2c_i801 Oct 25 11:23:53 snd_page_alloc Oct 25 11:23:53 dcdbas Oct 25 11:23:53 e1000e Oct 25 11:23:53 mfd_core Oct 25 11:23:53 usb_storage Oct 25 11:23:53 nouveau Oct 25 11:23:53 ttm Oct 25 11:23:53 drm_kms_helper Oct 25 11:23:53 drm Oct 25 11:23:53 i2c_algo_bit Oct 25 11:23:53 i2c_core Oct 25 11:23:53 mxm_wmi Oct 25 11:23:53 video Oct 25 11:23:53 wmi Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] CPU 1 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Pid: 7299, comm: kworker/1:8H Tainted: G D 3.6.2-1.fc16.x86_64 #1 Dell Inc. Inspiron 530/0RY007 Oct 25 11:23:53 dellix [ 846.824001] RIP: 0010:[<ffffffff81080421>] Oct 25 11:23:53 [<ffffffff81080421>] kthread_data+0x11/0x20 Oct 25 11:23:53 dellix [ 846.824001] RSP: 0018:ffff88003cf43940 EFLAGS: 00010096 Oct 25 11:23:53 dellix [ 846.824001] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001 Oct 25 11:23:53 dellix [ 846.824001] RDX: ffff8800cbc05000 RSI: 0000000000000001 RDI: ffff8800274fdc80 Oct 25 11:23:53 dellix [ 846.824001] RBP: ffff88003cf43958 R08: ffff8800274fdcf0 R09: 0000000000000001 Oct 25 11:23:53 dellix [ 846.824001] R10: 0000000000000000 R11: 000000000000002f R12: 0000000000000001 Oct 25 11:23:53 dellix [ 846.824001] R13: ffff8800274fe050 R14: ffff88003cf43748 R15: ffff8800274fdf58 Oct 25 11:23:53 dellix [ 846.824001] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000 Oct 25 11:23:53 dellix [ 846.824001] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Oct 25 11:23:53 dellix [ 846.824001] CR2: fffffffffffffff8 CR3: 0000000069b83000 CR4: 00000000000007e0 Oct 25 11:23:53 dellix [ 846.824001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 25 11:23:53 dellix [ 846.824001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 25 11:23:53 dellix [ 846.824001] Process kworker/1:8H (pid: 7299, threadinfo ffff88003cf42000, task ffff8800274fdc80) Oct 25 11:23:53 dellix [ 846.824001] Stack: Oct 25 11:23:53 dellix [ 846.824001] ffffffff8107b9a5 Oct 25 11:23:53 ffff88003cf43958 Oct 25 11:23:53 ffff8800cfc93d40 Oct 25 11:23:53 ffff88003cf439d8 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] ffffffff81617b2f Oct 25 11:23:53 ffff88003cf43998 Oct 25 11:23:53 ffffffff812baf42 Oct 25 11:23:53 ffff8800274fdc80 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] ffff88003cf43fd8 Oct 25 11:23:53 ffff88003cf43fd8 Oct 25 11:23:53 ffff88003cf43fd8 Oct 25 11:23:53 ffff88003cf439c8 Oct 25 11:23:53 dellix Oct 25 11:23:53 dellix [ 846.824001] Call Trace: Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8107b9a5>] ? wq_worker_sleeping+0x15/0xc0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81617b2f>] __schedule+0x5df/0x7c0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff812baf42>] ? put_io_context+0x72/0xa0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81618019>] schedule+0x29/0x70 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff810628cd>] do_exit+0x58d/0x8d0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8160e90a>] ? printk+0x61/0x63 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8161a23c>] oops_end+0xac/0xf0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff810177f8>] die+0x58/0x90 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81619b14>] do_trap+0xc4/0x170 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81014e85>] do_invalid_op+0x95/0xb0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0416cc8>] ? inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0432210>] ? gfs2_control_func+0x3e0/0x3e0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff811385b2>] ? pagevec_lookup+0x22/0x30 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04317f0>] ? gdlm_lock+0x330/0x330 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0432210>] ? gfs2_control_func+0x3e0/0x3e0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff816220bb>] invalid_op+0x1b/0x20 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0416cc8>] ? inode_go_sync+0x108/0x140 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0414c76>] do_xmote+0xc6/0x270 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa0415468>] run_queue+0x148/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa041460e>] ? gfs2_glock_put+0x4e/0x190 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa04155c0>] ? run_queue+0x2a0/0x2a0 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffffa041562d>] glock_work_func+0x6d/0x150 [gfs2] Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81079c7a>] process_one_work+0x13a/0x530 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8107b07e>] worker_thread+0x18e/0x510 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff8107aef0>] ? manage_workers+0x310/0x310 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81080153>] kthread+0x93/0xa0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81622244>] kernel_thread_helper+0x4/0x10 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff810800c0>] ? flush_kthread_worker+0xb0/0xb0 Oct 25 11:23:53 dellix [ 846.824001] [<ffffffff81622240>] ? gs_change+0x13/0x13 Oct 25 11:23:53 dellix [ 846.824001] Code: Oct 25 11:23:53 dellix 81 Oct 25 11:23:53 dellix e8 Oct 25 11:23:53 dellix e4 Oct 25 11:23:53 dellix c2 Oct 25 11:23:53 dellix fd Oct 25 11:23:53 dellix ff Oct 25 11:23:53 dellix e9 Fix posted to 3.7: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=3d1626889a64bd5a661544d582036a0a02104a60 Fix also posted to the 3.6 stable tree. Fix has been added to the 3.6 stable tree, so it should appear in the next stable tree release. Now in the 3.6.8 stable tree |