*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.11.1-200.fc19.  Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.

This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 2 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.

------- Comment From iranna.ankad.com 2013-10-11 09:18 EDT-------
This bug is still reproducible on Fedora20 Alpha.

FYI..

Fedora release 20 (Heisenbug)
Kernel 3.11.0-300.fc20.ppc64p7 on an ppc64 (hvc0)

jupiterioc-lp3 login: [ 4204.289042] EXT4-fs warning (device dm-0): ext4_da_update_reserve_space:362: ino 784618, allocated 1 with only 0 reserved metadata blocks (releasing 13 blocks with reserved 93 data blocks)
[ 4204.289197] ------------[ cut here ]------------
[ 4204.289216] WARNING: at fs/ext4/inode.c:363
[ 4204.289229] Modules linked in: nls_utf8 isofs loop nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables xts gf128mul dm_crypt pseries_rng nx_crypto ibmveth xfs libcrc32c ibmvscsi scsi_transport_srp scsi_tgt
[ 4204.289497] CPU: 157 PID: 12301 Comm: fsstress Not tainted 3.11.0-300.fc20.ppc64p7 #1
[ 4204.289520] task: c000000b62e20000 ti: c000000b5b000000 task.ti: c000000b5b000000
[ 4204.289536] NIP: c0000000002f9f20 LR: c0000000002f9f1c CTR: 00000000015c5f10
[ 4204.289553] REGS: c000000b5b003190 TRAP: 0700   Not tainted  (3.11.0-300.fc20.ppc64p7)
[ 4204.289575] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28222424  XER: 00000001
[ 4204.289652] SOFTE: 1
[ 4204.289661] CFAR: c00000000031ea64
[ 4204.298956]
GPR00: c0000000002f9f1c c000000b5b003410 c0000000012f7ee0 00000000000000b0
GPR04: c000000002aa27f8 c000000002ab3200 800000000666c500 0000000000000000
GPR08: c000000000c37ee0 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000028222422 c00000000ed07400 0000000000000000 c000000a831380b0
GPR16: 00000000100084b0 0000000000000060 0000000000000200 0000000000000001
GPR20: 000000000000000d 000000000013feb0 000000000000000d 0000000000000000
GPR24: c000000b0e9427f0 c000000c16761000 0000000000000025 c000000c16761000
GPR28: 0000000000000001 000000000000000d 000000000000000d c000000a831380b0
[ 4204.308192] NIP [c0000000002f9f20] .ext4_da_update_reserve_space+0x2c0/0x2e0
[ 4204.308279] LR [c0000000002f9f1c] .ext4_da_update_reserve_space+0x2bc/0x2e0
[ 4204.308339] Call Trace:
[ 4204.308401] [c000000b5b003410] [c0000000002f9f1c] .ext4_da_update_reserve_space+0x2bc/0x2e0 (unreliable)
[ 4204.308499] [c000000b5b0034d0] [c00000000032e8a8] .ext4_ext_map_blocks+0xf98/0x1760
[ 4204.308575] [c000000b5b003610] [c0000000002fa2d4] .ext4_map_blocks+0x394/0x5e0
[ 4204.308637] [c000000b5b0036f0] [c0000000002fdf80] .ext4_writepages+0x610/0xd00
[ 4204.319042] [c000000b5b003890] [c0000000001cb60c] .do_writepages+0x5c/0xc0
[ 4204.319124] [c000000b5b003910] [c0000000001bca28] .__filemap_fdatawrite_range+0x78/0xa0
[ 4204.319217] [c000000b5b0039d0] [c0000000001bcc00] .filemap_write_and_wait_range+0x50/0xc0
[ 4204.319259] [c000000b5b003a60] [c0000000001bd00c] .generic_file_aio_read+0x39c/0x780
[ 4204.319365] [c000000b5b003ba0] [c000000000241f1c] .do_sync_read+0x9c/0x110
[ 4204.329003] [c000000b5b003cf0] [c00000000024293c] .vfs_read+0xcc/0x200
[ 4204.329103] [c000000b5b003d90] [c000000000243774] .SyS_read+0x64/0xe0
[ 4204.329173] [c000000b5b003e30] [c000000000009dd4] syscall_exit+0x0/0x98
[ 4204.329188] Instruction dump:
[ 4204.329250] 4bfffe3c f9410070 3c82ff58 3cc2ff7b 38843f98 38a0016a 38c6c558 7faaeb78
[ 4204.329363] e87f0028 e8ff0040 48024ab9 60000000 <0fe00000> 813f02a8 815f02a4 913f02ac
[ 4204.339085] ---[ end trace 50ca4eae5f25cf88 ]---

Now that we have metadata reservation this should not be a huge problem even when we approach ENOSPC since we should be able to allocate metadata block even though it was not accounted for (hence the warning).

Since this is based on 3.11 kernel most of the fixes in that are should be already in except for the latest one from Jan Kara:

7d7345322d60edb0fa49a64a89b31360f01d09cb ext4: fix warning in ext4_da_update_reserve_space()

which is likely to fix this issue.

However I have no information about the workload what so ever, so I will not be able to reproduce it until I have more information about the test used to hit this.

-Lukas

------- Comment From iranna.ankad.com 2013-10-15 15:03 EDT-------
(In reply to comment #13)
> Now that we have metadata reservation this should not be a huge problem even
> when we approach ENOSPC since we should be able to allocate metadata block
> even though it was not accounted for (hence the warning).
>
> Since this is based on 3.11 kernel most of the fixes in that are should be
> already in except for the latest one from Jan Kara:
>
> 7d7345322d60edb0fa49a64a89b31360f01d09cb ext4: fix warning in
> ext4_da_update_reserve_space()
>
> which is likely to fix this issue.
>
> However I have no information about the workload what so ever, so I will not
> be able to reproduce it until I have more information about the test used to
> hit this.
>
> -Lukas

Please find below steps to reproduce this issue:

1. Install LTP (https://sourceforge.net/projects/ltp/files/LTP%20Source/ltp-20130904/ltp-full-20130904.tar.xz/download)
2. Compile LTP and run fsstress tests from an EXT4 partition/directory.
# cd ltp-full-20130904/testcases/kernel/fs/fsstress/
# nohup ./fsstress -d /home/EXT4 -l 0 -n 1000 -p 1000 -r &

This issue should be resolved with upstream commits

1c8924eb106c1ac755d5d35ce9b3ff42e89e2511 quota: provide interface for readding allocated space into reserved space
7d7345322d60edb0fa49a64a89b31360f01d09cb ext4: fix warning in ext4_da_update_reserve_space()

Thanks!
-Lukas

(In reply to Lukáš Czerner from comment #6)
> This issue should be resolved with upstream commits
> 
> 1c8924eb106c1ac755d5d35ce9b3ff42e89e2511 quota: provide interface for
> readding allocated space into reserved space
> 7d7345322d60edb0fa49a64a89b31360f01d09cb ext4: fix warning in
> ext4_da_update_reserve_space()

Should those two be sent to the upstream stable kernel?

I do not think that this is -stable material.

- The change is quite big for -stable
- The warning does not pose any real threat and there are couple of reasons why.   First of all since 3.9 we have something I called metadata reservation space which serves as a pool of blocks for the case like this when we underestimate the number of needed metadata blocks so we do not fail on writeback when the file system is in ENOSPC condition. Secondly even without the reservation space this will become a problem *only* in the case that the file system is in ENOSPC condition on writeback. So I do not think it's critical enough to get into stable.

Thanks!
-Lukas

Thanks Lukáš.  Based on that and the fact that the commits will be in the 3.12 release, I'm going to close this as UPSTREAM.  F19 will rebase to 3.12 not long after it is released, so we'll pick up the fixes then.

------- Comment From shubgoya.com 2013-10-24 08:22 EDT-------
Am getting below call traces & warning on latest F20 Alpha release. This is coming in '/var/log/messages' while running LTP. Can someone please check if it deserves a different bugzilla or can be accommodated in this bug?

Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939642] EXT4-fs warning (device dm-1): ext4_da_release_space:1330: ext4_da_release_space: ino 1706562, to_free 1 with only 0 reserved data blocks
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939685] ------------[ cut here ]------------
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939690] WARNING: at fs/ext4/inode.c:1331
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939694] Modules linked in: tun loop ip6table_filter ip6_tables ebtable_nat ebtables bnep bluetooth rfkill windfarm_smu_sat i2c_core ibmveth windfarm_pid nfsd auth_rpcgss oid_registry nfs_acl lockd sunrpc ibmvscsi scsi_transport_srp scsi_tgt
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939735] CPU: 0 PID: 53786 Comm: growfiles Not tainted 3.11.0-300.fc20.ppc64 #1
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939741] task: c00000045fa476c0 ti: c000000448430000 task.ti: c000000448430000
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939746] NIP: c00000000030917c LR: c000000000309178 CTR: 00000000015c5f10
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939752] REGS: c0000004484335d0 TRAP: 0700   Not tainted  (3.11.0-300.fc20.ppc64)
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939757] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 24004424  XER: 0000000b
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939771] SOFTE: 1
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939774] CFAR: c00000000032e0f4
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777]
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR00: c000000000309178 c000000448433850 c00000000133c030 0000000000000089
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR04: c0000000017027f8 c000000001713200 000000000000000b 0000000000000000
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR08: c000000000c5c030 0000000000000000 0000000000000000 0000000000000000
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR12: 0000000024004422 c000000007f00000 00000000100318d0 0000000000000005
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR16: 0000000000000000 ffffffffffffffff 0000000000000001 000000001000dfe8
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR20: 0000000000001000 c0000004efb92800 0000000000000000 c0000004b3260fe8
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR24: 0000000000000000 000000000000f000 0000000000001000 c0000004efb92800
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939777] GPR28: f0000000010b9830 0000000000000001 c0000004b3260cb0 0000000000000001
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939845] NIP [c00000000030917c] .ext4_da_invalidatepage+0x38c/0x3b0
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939851] LR [c000000000309178] .ext4_da_invalidatepage+0x388/0x3b0
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939855] Call Trace:
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939860] [c000000448433850] [c000000000309178] .ext4_da_invalidatepage+0x388/0x3b0 (unreliable)
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939869] [c000000448433940] [c0000000001df05c] .truncate_inode_pages_range+0x63c/0x700
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939876] [c000000448433a90] [c0000000001df1c0] .truncate_pagecache+0x60/0xa0
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939883] [c000000448433b20] [c000000000310518] .ext4_setattr+0x548/0x7e0
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939890] [c000000448433bf0] [c000000000278404] .notify_change+0x294/0x4b0
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939897] [c000000448433ca0] [c00000000024f0ec] .do_truncate+0x8c/0x100
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939904] [c000000448433d80] [c00000000024f658] .do_sys_ftruncate.constprop.12+0x1b8/0x230
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939912] [c000000448433e30] [c000000000009ed4] syscall_exit+0x0/0x98
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939916] Instruction dump:
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939920] 7fa5eb78 4bfffe28 e87e0028 e8fe0040 3c82ff55 3cc2ff78 38840130 38a00532
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939931] 38c61f08 39000001 48024eed 60000000 <0fe00000> 813e02a4 7d3207b4 7d3f4b78
Oct 23 08:50:13 jupiterioc-lp2 kernel: [525675.939942] ---[ end trace f54d3ad7aa2758f0 ]---

------- Comment From onmahaja.com 2013-10-24 08:33 EDT-------
Shubham,  Query regarding your comment #19 :
(1) Is this trace produced from same test (file system IO stress tests) & LTP version ?
(2) 'uname -a' o/p ?

------- Comment From onmahaja.com 2013-10-24 08:36 EDT-------
Does LTP does it provide some kind of verbose logging information for us to know under what workload we are hitting this crash. If so , I would request you to reproduce the tests with most verbose level of logging.

------- Comment From shubgoya.com 2013-10-24 08:50 EDT-------
(In reply to comment #20)
> Shubham,  Query regarding your comment #19 :
> (1) Is this trace produced from same test (file system IO stress tests) &
> LTP version ?
> (2) 'uname -a' o/p ?

Onkar,

The original bug was coming while running fsstress & my bug (comment #19) is coming while running LTP. Should I open a new bug or keep it here only?

Thanks...

------- Comment From onmahaja.com 2013-10-24 08:52 EDT-------
[root@jupiterioc-lp2 ~]# uname -a
Linux jupiterioc-lp2.austin.ibm.com 3.11.0-300.fc20.ppc64 #1 SMP Thu Sep 5 16:13:52 MST 2013 ppc64 ppc64 ppc64 GNU/Linux

Please open a new bug, this is different issue entirely.

Thanks!
-Lukas

------- Comment From shubgoya.com 2013-10-24 12:04 EDT-------
(In reply to comment #27)
> Please open a new bug, this is different issue entirely.

Okay done. Thanks for your suggestion....

------- Comment From iranna.ankad.com 2013-11-13 17:33 EDT-------
(In reply to comment #18)
> Thanks Luk??.  Based on that and the fact that the commits will be in the
> 3.12 release, I'm going to close this as UPSTREAM.  F19 will rebase to 3.12
> not long after it is released, so we'll pick up the fixes then.

Yeah..looks like commits are not yet available on Fedora20 Beta, I could still recreate this issue.

FYI,

Kernel 3.11.6-301.fc20.ppc64p7 on an ppc64 (hvc0)

als0153 login: [ 7343.105816] EXT4-fs warning (device dm-0): ext4_da_update_reserve_space:362: ino 263873, allocated 1 with only 0 reserved metadata blocks (releasing 114 blocks with reserved 114 data blocks)
[ 7343.105864] ------------[ cut here ]------------
[ 7343.105869] WARNING: at fs/ext4/inode.c:363
[ 7343.105872] Modules linked in: rfcomm scsi_transport_iscsi hidp l2tp_ppp l2tp_netlink l2tp_core nfnetlink ipt_ULOG pppoe pppox ppp_generic slhc af_key irda crc_ccitt rose appletalk af_802154 atm nfc netrom ax25 ipx p8022 p8023 psnap llc ebtable_nat nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE bnep bluetooth rfkill ip6table_mangle ip6table_security ip6table_raw ip6t_REJECT iptable_nat nf_nat_ipv4 iptable_mangle iptable_security iptable_raw nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ebtable_filter ebtables ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables xts gf128mul dm_crypt ibmveth nfsd auth_rpcgss oid_registry nfs_acl lockd sunrpc xfs libcrc32c ibmvscsi scsi_transport_srp scsi_tgt
[ 7343.105995] CPU: 4 PID: 14484 Comm: kworker/u33:0 Not tainted 3.11.6-301.fc20.ppc64p7 #1
[ 7343.106007] Workqueue: writeback .bdi_writeback_workfn (flush-253:0)
[ 7343.106014] task: c0000000a80d1120 ti: c000000002b58000 task.ti: c000000002b58000
[ 7343.106019] NIP: c0000000002fa3a0 LR: c0000000002fa39c CTR: 000000000168a440
[ 7343.106025] REGS: c000000002b5afb0 TRAP: 0700   Not tainted  (3.11.6-301.fc20.ppc64p7)
[ 7343.106030] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28c52524  XER: 00000003
[ 7343.106044] SOFTE: 1
[ 7343.106046] CFAR: c00000000031eea4
[ 7343.106050]
GPR00: c0000000002fa39c c000000002b5b230 c0000000012f7ed0 00000000000000b2
GPR04: c0000000017827f8 c000000001793200 0000000000004000 0000000000010000
GPR08: c000000000c37ed0 0000000000000000 0000000000000000 0000000000000080
GPR12: 0000000028c52522 c00000000ed81000 c0000000000c8290 c000000211ad3cb0
GPR16: 0000000000000001 00000000000001e3 0000000000000008 0000000000000001
GPR20: 0000000000000072 0000000000247797 0000000000000072 0000000000000000
GPR24: c0000003fad0bbb0 c0000003f9679000 0000000000000025 c0000003f9679000
GPR28: 0000000000000001 0000000000000072 0000000000000072 c000000211ad3cb0
[ 7343.106118] NIP [c0000000002fa3a0] .ext4_da_update_reserve_space+0x2c0/0x2e0
[ 7343.106124] LR [c0000000002fa39c] .ext4_da_update_reserve_space+0x2bc/0x2e0
[ 7343.106128] Call Trace:
[ 7343.106132] [c000000002b5b230] [c0000000002fa39c] .ext4_da_update_reserve_space+0x2bc/0x2e0 (unreliable)
[ 7343.106142] [c000000002b5b2f0] [c00000000032ecd4] .ext4_ext_map_blocks+0xf84/0x1750
[ 7343.106149] [c000000002b5b430] [c0000000002fa754] .ext4_map_blocks+0x394/0x5e0
[ 7343.106155] [c000000002b5b510] [c0000000002fe400] .ext4_writepages+0x610/0xd00
[ 7343.106163] [c000000002b5b6b0] [c0000000001cb64c] .do_writepages+0x5c/0xc0
[ 7343.106170] [c000000002b5b730] [c00000000027cbe0] .__writeback_single_inode+0x60/0x380
[ 7343.106176] [c000000002b5b7d0] [c00000000027d1a8] .writeback_sb_inodes+0x2a8/0x4d0
[ 7343.106182] [c000000002b5b8f0] [c00000000027d4b4] .__writeback_inodes_wb+0xe4/0x150
[ 7343.106189] [c000000002b5b9a0] [c00000000027e48c] .wb_writeback+0x2fc/0x3d0
[ 7343.106194] [c000000002b5ba90] [c00000000027ef08] .bdi_writeback_workfn+0x138/0x4d0
[ 7343.106203] [c000000002b5bbc0] [c0000000000bdf84] .process_one_work+0x1a4/0x4d0
[ 7343.106209] [c000000002b5bc60] [c0000000000bec4c] .worker_thread+0x17c/0x490
[ 7343.106216] [c000000002b5bd30] [c0000000000c8380] .kthread+0xf0/0x100
[ 7343.106223] [c000000002b5be30] [c00000000000a0dc] .ret_from_kernel_thread+0x5c/0x80
[ 7343.106229] Instruction dump:
[ 7343.106232] 4bfffe3c f9410070 3c82ff58 3cc2ff7b 38843fa8 38a0016a 38c6c968 7faaeb78
[ 7343.106243] e87f0028 e8ff0040 48024a79 60000000 <0fe00000> 813f02a8 815f02a4 913f02ac
[ 7343.106254] ---[ end trace cae5b4fb78e9c000 ]---

------- Comment From kamaleshb.com 2013-12-12 06:11 EDT-------
*** Bug 100869 has been marked as a duplicate of this bug. ***

------- Comment From onmahaja.com 2013-12-16 08:58 EDT-------
(In reply to comment #29)
> (In reply to comment #18)
> > Thanks Luk??.  Based on that and the fact that the commits will be in the
> > 3.12 release, I'm going to close this as UPSTREAM.  F19 will rebase to 3.12
> > not long after it is released, so we'll pick up the fixes then.
> Yeah..looks like commits are not yet available on Fedora20 Beta, I could
> still recreate this issue.
>
>
> FYI,
>
> Kernel 3.11.6-301.fc20.ppc64p7 on an ppc64 (hvc0)
>
> als0153 login: [ 7343.105816] EXT4-fs warning (device dm-0):
> ext4_da_update_reserve_space:362: ino 263873, allocated 1 with only 0
> reserved metadata blocks (releasing 114 blocks with reserved 114 data blocks)
> [ 7343.105864] ------------[ cut here ]------------
> [ 7343.105869] WARNING: at fs/ext4/inode.c:363
> [ 7343.105872] Modules linked in: rfcomm scsi_transport_iscsi hidp l2tp_ppp
> l2tp_netlink l2tp_core nfnetlink ipt_ULOG pppoe pppox ppp_generic slhc
> af_key irda crc_ccitt rose appletalk af_802154 atm nfc netrom ax25 ipx p8022
> p8023 psnap llc ebtable_nat nf_conntrack_netbios_ns nf_conntrack_broadcast
> ipt_MASQUERADE bnep bluetooth rfkill ip6table_mangle ip6table_security
> ip6table_raw ip6t_REJECT iptable_nat nf_nat_ipv4 iptable_mangle
> iptable_security iptable_raw nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
> ebtable_filter ebtables ip6table_filter ip6table_nat nf_conntrack_ipv6
> nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables xts gf128mul
> dm_crypt ibmveth nfsd auth_rpcgss oid_registry nfs_acl lockd sunrpc xfs
> libcrc32c ibmvscsi scsi_transport_srp scsi_tgt
> [ 7343.105995] CPU: 4 PID: 14484 Comm: kworker/u33:0 Not tainted
> 3.11.6-301.fc20.ppc64p7 #1
> [ 7343.106007] Workqueue: writeback .bdi_writeback_workfn (flush-253:0)
> [ 7343.106014] task: c0000000a80d1120 ti: c000000002b58000 task.ti:
> c000000002b58000
> [ 7343.106019] NIP: c0000000002fa3a0 LR: c0000000002fa39c CTR:
> 000000000168a440
> [ 7343.106025] REGS: c000000002b5afb0 TRAP: 0700   Not tainted
> (3.11.6-301.fc20.ppc64p7)
> [ 7343.106030] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28c52524  XER:
> 00000003
> [ 7343.106044] SOFTE: 1
> [ 7343.106046] CFAR: c00000000031eea4
> [ 7343.106050]
> GPR00: c0000000002fa39c c000000002b5b230 c0000000012f7ed0 00000000000000b2
> GPR04: c0000000017827f8 c000000001793200 0000000000004000 0000000000010000
> GPR08: c000000000c37ed0 0000000000000000 0000000000000000 0000000000000080
> GPR12: 0000000028c52522 c00000000ed81000 c0000000000c8290 c000000211ad3cb0
> GPR16: 0000000000000001 00000000000001e3 0000000000000008 0000000000000001
> GPR20: 0000000000000072 0000000000247797 0000000000000072 0000000000000000
> GPR24: c0000003fad0bbb0 c0000003f9679000 0000000000000025 c0000003f9679000
> GPR28: 0000000000000001 0000000000000072 0000000000000072 c000000211ad3cb0
> [ 7343.106118] NIP [c0000000002fa3a0]
> .ext4_da_update_reserve_space+0x2c0/0x2e0
> [ 7343.106124] LR [c0000000002fa39c]
> .ext4_da_update_reserve_space+0x2bc/0x2e0
> [ 7343.106128] Call Trace:
> [ 7343.106132] [c000000002b5b230] [c0000000002fa39c]
> .ext4_da_update_reserve_space+0x2bc/0x2e0 (unreliable)
> [ 7343.106142] [c000000002b5b2f0] [c00000000032ecd4]
> .ext4_ext_map_blocks+0xf84/0x1750
> [ 7343.106149] [c000000002b5b430] [c0000000002fa754]
> .ext4_map_blocks+0x394/0x5e0
> [ 7343.106155] [c000000002b5b510] [c0000000002fe400]
> .ext4_writepages+0x610/0xd00
> [ 7343.106163] [c000000002b5b6b0] [c0000000001cb64c] .do_writepages+0x5c/0xc0
> [ 7343.106170] [c000000002b5b730] [c00000000027cbe0]
> .__writeback_single_inode+0x60/0x380
> [ 7343.106176] [c000000002b5b7d0] [c00000000027d1a8]
> .writeback_sb_inodes+0x2a8/0x4d0
> [ 7343.106182] [c000000002b5b8f0] [c00000000027d4b4]
> .__writeback_inodes_wb+0xe4/0x150
> [ 7343.106189] [c000000002b5b9a0] [c00000000027e48c]
> .wb_writeback+0x2fc/0x3d0
> [ 7343.106194] [c000000002b5ba90] [c00000000027ef08]
> .bdi_writeback_workfn+0x138/0x4d0
> [ 7343.106203] [c000000002b5bbc0] [c0000000000bdf84]
> .process_one_work+0x1a4/0x4d0
> [ 7343.106209] [c000000002b5bc60] [c0000000000bec4c]
> .worker_thread+0x17c/0x490
> [ 7343.106216] [c000000002b5bd30] [c0000000000c8380] .kthread+0xf0/0x100
> [ 7343.106223] [c000000002b5be30] [c00000000000a0dc]
> .ret_from_kernel_thread+0x5c/0x80
> [ 7343.106229] Instruction dump:
> [ 7343.106232] 4bfffe3c f9410070 3c82ff58 3cc2ff7b 38843fa8 38a0016a
> 38c6c968 7faaeb78
> [ 7343.106243] e87f0028 e8ff0040 48024a79 60000000 <0fe00000> 813f02a8
> 815f02a4 913f02ac
> [ 7343.106254] ---[ end trace cae5b4fb78e9c000 ]---

What tests re-created this crash , if possible please attach the test logs - this might greatly simplify things to trace the problem ?

Please also attach SOS report created on the system in question *after* the tests were conducted ...

Thx,
Onkar