The problem is that the delete workqueue should never be scheduled for read only filesystems. Should be fairly simple to fix.

Created attachment 487303 [details]
Upstream patch

This patch should fix the problem. There are two places that the delete_workqueue is scheduled. One is when we are searching for new blocks to allocate, so that will never happen on a read-only/spectator mount. The other is when we receive an iopen callback from a remote node, and that is the code path that is fixed here.

This needs to be fixed for all read-only mounts and not just spectator mounts.

I applied the patch and rebuilt the kernel... I just got this panic after installing it and running for a while. I'm not sure if it's part of the same issue or not though.

kernel: Kernel BUG at fs/gfs2/trans.c:35
kernel: invalid opcode: 0000 [1] SMP 
kernel: last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:07:00.1/host3/rport-3:0-7/target3:0:5/3:0:5:2/timeout
kernel: CPU 3 
kernel: Modules linked in: ip_conntrack_ftp xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables ipv6 xfrm_nalgo crypto_api lock_dlm gfs2 dlm configfs dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom shpchp hpilo serio_raw pcspkr bnx2 sg dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd 

kernel: Pid: 5500, comm: httpd.worker Not tainted 2.6.18-194.32.1.ubi.el5 #1 
kernel: RIP: 0010:[<ffffffff8842b5a4>]  [<ffffffff8842b5a4>] :gfs2:gfs2_do_trans_begin+0x3d/0x144
kernel: RSP: 0018:ffff81037293dc08  EFLAGS: 00010246
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001 
kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff81060f2f5000 
kernel: RBP: ffff8104db5ef368 R08: ffff81037293c000 R09: 0000000000000246 
kernel: R10: ffff8106350ead90 R11: 0000000000000000 R12: 0000000000000000 
kernel: R13: ffff81061f007ac0 R14: ffff81060f2f5000 R15: 0000000000000001 
kernel: FS:  00002aaadfdff940(0000) GS:ffff810c1ff376c0(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: CR2: 00002aab5c595000 CR3: 000000037993b000 CR4: 00000000000006e0 
kernel: Process httpd.worker (pid: 5500, threadinfo ffff81037293c000, task ffff81060df4a080)
kernel: Stack:  ffff81037293dd48 ffff8104db5ef5d0 ffff8104db5ef368 ffff81060f2f5000
kernel: ffff81061f007ac0 ffff81037293de48 ffff81037293dd48 ffffffff88423f25
kernel: ffff8104db47a7c8 ffff8104db47a7c8 ffff8104db47a778 000000010000157c
kernel: Call Trace:
kernel: [<ffffffff88423f25>] :gfs2:gfs2_delete_inode+0x129/0x1b4
kernel: [<ffffffff88423e39>] :gfs2:gfs2_delete_inode+0x3d/0x1b4
kernel: [<ffffffff88423dfc>] :gfs2:gfs2_delete_inode+0x0/0x1b4
kernel: [<ffffffff8002f438>] generic_delete_inode+0xc6/0x143
kernel: [<ffffffff8000d544>] dput+0xf6/0x114
kernel: [<ffffffff8000d09e>] do_lookup+0x1ac/0x1e6
kernel: [<ffffffff8000a29c>] __link_path_walk+0xa01/0xf5b
kernel: [<ffffffff8000ea4b>] link_path_walk+0x42/0xb2
kernel: [<ffffffff8000cd72>] do_path_lookup+0x275/0x2f1
kernel: [<ffffffff80012851>] getname+0x15b/0x1c2
kernel: [<ffffffff800239d1>] __user_walk_fd+0x37/0x4c
kernel: [<ffffffff80028905>] vfs_stat_fd+0x1b/0x4a
kernel: [<ffffffff80023703>] sys_newstat+0x19/0x31
kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0
kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
kernel:
kernel:
kernel: Code: 0f 0b 68 c5 02 43 88 c2 23 00 48 8b 3d 23 cf ee f7 be 50 00 
kernel: RIP  [<ffffffff8842b5a4>] :gfs2:gfs2_do_trans_begin+0x3d/0x144
kernel: RSP <ffff81037293dc08>

Hmm, I think I can see what is going on here now. Looks like I need to expand the scope of the patch...

Created attachment 488150 [details]
Updated patch

This is an upstream patch, the RHEL5 one will be slightly different.

Created attachment 488152 [details]
RHEL5 version of patch

RHEL5 version of the updated patch.

Applied RHEL5 patch, this panic looks similar though...



kernel: Kernel BUG at fs/gfs2/trans.c:35
kernel: invalid opcode: 0000 [2] SMP 
kernel: last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:07:00.1/host3/rport-3:0-7/target3:0:5/3:0:5:2/timeout
kernel: CPU 11 
kernel: Modules linked in: ip_conntrack_ftp xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables ipv6 xfrm_nalgo crypto_api lock_dlm gfs2 dlm configfs dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom shpchp bnx2 hpilo pcspkr serio_raw sg dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
kernel: Pid: 23225, comm: httpd.worker Not tainted 2.6.18-194.32.1ubi1.el5 #1
kernel: RIP: 0010:[<ffffffff8842d594>]  [<ffffffff8842d594>] :gfs2:gfs2_do_trans_begin+0x3d/0x144
kernel: RSP: 0018:ffff81026e2b5c08  EFLAGS: 00010246
kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff81061dd51000
kernel: RBP: ffff810b5a3f1368 R08: ffff81026e2b4000 R09: 0000000000000246
kernel: R10: ffff81063510c790 R11: ffff810b5719b7c0 R12: 0000000000000000
kernel: R13: ffff81061f038bc0 R14: ffff81061dd51000 R15: 0000000000000001
kernel: FS:  00002aaad0de7940(0000) GS:ffff810c1fc27cc0(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: CR2: 00002aab5c47f000 CR3: 00000001c3e41000 CR4: 00000000000006e0
kernel: Process httpd.worker (pid: 23225, threadinfo ffff81026e2b4000, task ffff81024cdfc0c0)
kernel: Stack:  ffff81026e2b5d48 ffff810b5a3f15d0 ffff810b5a3f1368 ffff81061dd51000
kernel: ffff81061f038bc0 ffff81026e2b5e48 ffff81026e2b5d48 ffffffff88425f15
kernel: ffff810b59883128 ffff810b59883128 ffff810b598830d8 0000000100005ab9
kernel: Call Trace:
kernel: [<ffffffff88425f15>] :gfs2:gfs2_delete_inode+0x129/0x1b4
kernel: [<ffffffff88425e29>] :gfs2:gfs2_delete_inode+0x3d/0x1b4
kernel: [<ffffffff88425dec>] :gfs2:gfs2_delete_inode+0x0/0x1b4
kernel: [<ffffffff8002f438>] generic_delete_inode+0xc6/0x143
kernel: [<ffffffff8000d544>] dput+0xf6/0x114
kernel: [<ffffffff8000d09e>] do_lookup+0x1ac/0x1e6
kernel: [<ffffffff8000a29c>] __link_path_walk+0xa01/0xf5b
kernel: [<ffffffff8000ea4b>] link_path_walk+0x42/0xb2
kernel: [<ffffffff8000cd72>] do_path_lookup+0x275/0x2f1
kernel: [<ffffffff80012851>] getname+0x15b/0x1c2
kernel: [<ffffffff800239d1>] __user_walk_fd+0x37/0x4c
kernel: [<ffffffff80028905>] vfs_stat_fd+0x1b/0x4a
kernel: [<ffffffff80023703>] sys_newstat+0x19/0x31
kernel: [<ffffffff8005d229>] tracesys+0x71/0xe0
kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
kernel:
kernel:
kernel: Code: 0f 0b 68 c5 22 43 88 c2 23 00 48 8b 3d 33 af ee f7 be 50 00 
kernel: RIP  [<ffffffff8842d594>] :gfs2:gfs2_do_trans_begin+0x3d/0x144
kernel: RSP <ffff81026e2b5c08>

a GPF also, this might help...


kernel: <0>general protection fault: 0000 [3] SMP 
kernel: last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:07:00.1/host3/rport-3:0-7/target3:0:5/3:0:5:2/timeout
kernel: CPU 12 
kernel: Modules linked in: ip_conntrack_ftp xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables ipv6 xfrm_nalgo crypto_api lock_dlm gfs2 dlm configfs dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom shpchp bnx2 hpilo pcspkr serio_raw sg dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
kernel: Pid: 6018, comm: glock_workqueue Not tainted 2.6.18-194.32.1ubi1.el5 #1
kernel: RIP: 0010:[<ffffffff88417013>]  [<ffffffff88417013>] :gfs2:do_xmote+0x98/0x1aa
kernel: RSP: 0018:ffff810615b93de0  EFLAGS: 00010202
kernel: RAX: d6b4d24cca2a7d72 RBX: ffff810b598830d8 RCX: 000000000000000c
kernel: RDX: c76823ab7b67442d RSI: 0000000000000000 RDI: ffff81026e2b5c48
kernel: RBP: c76823ab7b67442d R08: ffff810615b92000 R09: 0000000000000033
kernel: R10: ffff810001037e10 R11: ffffffff88416c1c R12: 0000000000000000
kernel: R13: 0000000000000000 R14: ffffffff8842e240 R15: ffffffff88417a00
kernel: FS:  0000000000000000(0000) GS:ffff81061fc4a6c0(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
kernel: CR2: 00002b9d7ff6d000 CR3: 000000060fb18000 CR4: 00000000000006e0
kernel: Process glock_workqueue (pid: 6018, threadinfo ffff810615b92000, task ffff810c1e414040)
kernel: Stack:  ffff81061dd51000 ffff810b598830d8 0000000000000000 0000000000000000
kernel: 0000000000000282 ffff810b598830d8 ffffffff88417a67 ffffffff88417af9
kernel: ffff810b598831c0 ffff810b598831c8 ffff81061d42c240 ffffffff8004d743
kernel: Call Trace:
kernel: [<ffffffff88417a67>] :gfs2:glock_work_func+0x0/0xe2
kernel: [<ffffffff88417af9>] :gfs2:glock_work_func+0x92/0xe2
kernel: [<ffffffff8004d743>] run_workqueue+0x94/0xe4
kernel: [<ffffffff80049f78>] worker_thread+0x0/0x122
kernel: [<ffffffff800a0947>] keventd_create_kthread+0x0/0xc4
kernel: [<ffffffff8004a068>] worker_thread+0xf0/0x122
kernel: [<ffffffff8008d0ad>] default_wake_function+0x0/0xe
kernel: [<ffffffff800a0947>] keventd_create_kthread+0x0/0xc4
kernel: [<ffffffff8003296e>] kthread+0xfe/0x132
kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
kernel: [<ffffffff800a0947>] keventd_create_kthread+0x0/0xc4
kernel: [<ffffffff80032870>] kthread+0x0/0x132
kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
kernel:
kernel:
kernel: Code: 48 89 42 08 48 89 10 48 89 7f 08 48 89 3f e8 a5 ec ff ff 48 
kernel: RIP  [<ffffffff88417013>] :gfs2:do_xmote+0x98/0x1aa
kernel: RSP <ffff810615b93de0>

I agree comment #7 looks similar, but I'm rather confused as to how that can happen with the patch in place.... the patch bypasses almost all of the content of gfs2_delete_inode() and certainly the call to gfs2_trans_begin(). I know it is a silly question, but are you 100% sure that you booted the patched kernel?

Comment #8 looks like something completely different. Was this after you'd already hit the comment #7 problem, or was it on a fresh boot of the kernel?

The other thing we can try and check is to be sure that the sb has the MS_RDONLY flag set (this should be set when spectator is set) but I can't see how that could be missed at the moment.

Also, note that although gfs is provided as a kmod, gfs2 is provided by the kernel itself. If you have a gfs2 kmod floating around still, that will be (very) old code and it will also get loaded in preference to the kernel provided gfs2 if it exists on your system. So if you have a gfs2 kmod anywhere, you'll need to remove it by hand.

Comment #8 came immediately after comment #7, I added it as further information.

This is the newer kernel version, there are no kmods on the machine, and these are the only modules:


/lib/modules/2.6.18-194.32.1ubi1.el5/kernel/fs/gfs2/gfs2.ko
./lib/modules/2.6.18-194.32.1ubi1.el5/kernel/fs/gfs2/locking/dlm/lock_dlm.ko
./lib/modules/2.6.18-194.32.1ubi1.el5/kernel/fs/gfs2/locking/nolock/lock_nolock.ko
./lib/modules/2.6.18-194.32.1ubi1.el5/kernel/fs/configfs/configfs.ko

2.6.18-194.32.1ubi.el5 Used the first patch provided in comment #1
2.6.18-194.32.1ubi1.el5 Used the RHEL5 patch provided in comment #6

bollocks... you made me second-guess myself and i checked the specfile... patch was included but never applied. re-building the kernel and i'll get back to you.

Did that fix the issue?

No kernel panics since the patch applied. I would say yes, this fixes the problem.

Thanks for your help

Patch(es) available in kernel-2.6.18-257.el5
You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5
Detailed testing feedback is always welcomed.

I ran the test case against -238.el5 (rhel5.6 kernel) and was able to reproduce the panic.  Running the same test case on -262.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-1065.html