Bug 1255509 - BUG: unable to handle kernel paging request at ffffffffffffffd8
BUG: unable to handle kernel paging request at ffffffffffffffd8
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
22
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Kernel Maintainer List
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-20 15:12 EDT by Yasuo Ohgaki
Modified: 2015-10-29 07:32 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-29 07:32:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yasuo Ohgaki 2015-08-20 15:12:00 EDT
System hanged up next morning. Backintime and cron-daily was working while the kernel crash probably, so disk was very busy.

 8月 20 21:14:29 myfedora run-parts[21358]: (/etc/cron.daily) starting mlocate
 8月 20 21:25:31 myfedora kernel: general protection fault: 0000 [#1] SMP 

I'm not sure if this change has significance, but I got this crash after enabling btrfs' space_cache. 

It seems this is the first kernel panic for this system since last December.
(I replaced the host name to "myfedora")

$ sudo journalctl -a | grep "myfedora: general protection fault"
 8月 20 21:25:31 myfedora kernel: general protection fault: 0000 [#1] SMP 
 8月 20 21:25:31 myfedora kernel: general protection fault: 0000 [#2] SMP 

$ uname -a
Linux myfedora 4.1.5-200.fc22.x86_64 #1 SMP Mon Aug 10 23:38:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ lsmod
Module                  Size  Used by
bnep                   24576  2
bluetooth             491520  5 bnep
rfkill                 24576  2 bluetooth
xt_CHECKSUM            16384  1
iptable_mangle         16384  1
ipt_MASQUERADE         16384  3
nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
iptable_nat            16384  1
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 28672  2 nf_nat_ipv4,nf_nat_masquerade_ipv4
nf_conntrack_ipv4      16384  2
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
xt_conntrack           16384  1
nf_conntrack          106496  5 nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4
tun                    28672  1
bridge                114688  0
ebtable_filter         16384  0
ebtables               32768  1 ebtable_filter
ip6table_filter        16384  0
ip6_tables             28672  1 ip6table_filter
vmnet                  53248  16
ppdev                  20480  0
parport_pc             28672  0
parport                40960  2 ppdev,parport_pc
fuse                   94208  6
vmw_vsock_vmci_transport    32768  0
vsock                  36864  1 vmw_vsock_vmci_transport
vmw_vmci               69632  1 vmw_vsock_vmci_transport
vmmon                  90112  0
raid456                94208  1
intel_rapl             20480  0
async_raid6_recov      20480  1 raid456
iosf_mbi               16384  1 intel_rapl
snd_hda_codec_realtek    90112  1
async_memcpy           16384  2 raid456,async_raid6_recov
async_pq               16384  2 raid456,async_raid6_recov
x86_pkg_temp_thermal    16384  0
async_xor              16384  3 async_pq,raid456,async_raid6_recov
async_tx               16384  5 async_pq,raid456,async_xor,async_memcpy,async_raid6_recov
snd_hda_codec_generic    73728  1 snd_hda_codec_realtek
snd_hda_codec_hdmi     53248  1
coretemp               16384  0
kvm_intel             159744  0
kvm                   495616  1 kvm_intel
iTCO_wdt               16384  0
snd_hda_intel          32768  12
snd_hda_controller     32768  1 snd_hda_intel
iTCO_vendor_support    16384  1 iTCO_wdt
snd_hda_codec         110592  5 snd_hda_codec_realtek,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_intel,snd_hda_controller
snd_soc_rt5640         94208  0
joydev                 20480  0
mxm_wmi                16384  0
snd_soc_rl6231         16384  1 snd_soc_rt5640
crct10dif_pclmul       16384  0
snd_soc_core          188416  1 snd_soc_rt5640
crc32_pclmul           16384  0
snd_hda_core           32768  5 snd_hda_codec_realtek,snd_hda_codec_hdmi,snd_hda_codec_generic,snd_hda_codec,snd_hda_controller
ghash_clmulni_intel    16384  0
snd_compress           20480  1 snd_soc_core
snd_hwdep              16384  1 snd_hda_codec
snd_pcm_dmaengine      16384  1 snd_soc_core
snd_seq                69632  0
lpc_ich                24576  0
snd_seq_device         16384  1 snd_seq
mei_me                 24576  0
i2c_i801               20480  0
mfd_core               16384  1 lpc_ich
snd_pcm               118784  7 snd_soc_rt5640,snd_soc_core,snd_hda_codec_hdmi,snd_hda_codec,snd_hda_intel,snd_hda_controller,snd_pcm_dmaengine
mei                    90112  1 mei_me
shpchp                 40960  0
tpm_tis                20480  0
tpm                    36864  1 tpm_tis
wmi                    20480  1 mxm_wmi
snd_timer              32768  2 snd_pcm,snd_seq
regmap_i2c             16384  1 snd_soc_rt5640
snd                    77824  36 snd_hda_codec_realtek,snd_soc_core,snd_hwdep,snd_timer,snd_hda_codec_hdmi,snd_pcm,snd_seq,snd_hda_codec_generic,snd_hda_codec,snd_hda_intel,snd_seq_device,snd_compress
dw_dmac                16384  0
i2c_designware_platform    16384  0
acpi_pad               36864  0
i2c_designware_core    20480  1 i2c_designware_platform
dw_dmac_core           24576  1 dw_dmac
soundcore              16384  1 snd
snd_soc_sst_acpi       16384  0
nfsd                  323584  1
auth_rpcgss            65536  1 nfsd
nfs_acl                16384  1 nfsd
lockd                  94208  1 nfsd
grace                  16384  2 nfsd,lockd
sunrpc                315392  7 nfsd,auth_rpcgss,lockd,nfs_acl
ata_generic            16384  0
pata_acpi              16384  0
8021q                  32768  0
garp                   16384  1 8021q
stp                    16384  2 garp,bridge
llc                    16384  3 stp,garp,bridge
mrp                    20480  1 8021q
btrfs                 974848  4
xor                    24576  2 btrfs,async_xor
i915                 1097728  8
raid6_pq              106496  4 async_pq,raid456,btrfs,async_raid6_recov
i2c_algo_bit           16384  1 i915
drm_kms_helper        122880  1 i915
crc32c_intel           24576  1
drm                   331776  9 i915,drm_kms_helper
e1000e                241664  0
serio_raw              16384  0
uas                    24576  0
usb_storage            65536  3 uas
pata_marvell           16384  0
sdhci_acpi             16384  0
sdhci                  45056  1 sdhci_acpi
ptp                    20480  1 e1000e
pps_core               20480  1 ptp
mmc_core              126976  2 sdhci,sdhci_acpi
video                  24576  1 i915
i2c_hid                20480  0

$ mount | grep '^/dev'
/dev/sda3 on / type btrfs (rw,relatime,compress=lzo,ssd,space_cache)
/dev/sda3 on /var type btrfs (rw,relatime,compress=lzo,ssd,space_cache)
/dev/sda3 on /home type btrfs (rw,relatime,compress=lzo,ssd,space_cache)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)
/dev/sdh1 on /backup type btrfs (rw,relatime,compress=lzo,space_cache)
/dev/sdg1 on /backup/backintime type btrfs (rw,relatime,compress=lzo,space_cache)
/dev/md0 on /home/yohgaki/vmware type btrfs (rw,relatime,compress=lzo,space_cache)
/dev/md0 on /home/backup type btrfs (rw,relatime,compress=lzo,space_cache)
/dev/md0 on /home/yohgaki/workspace type btrfs (rw,relatime,compress=lzo,space_cache)
/dev/md0 on /usr/local type btrfs (rw,relatime,compress=lzo,space_cache)
/dev/md0 on /home/yohgaki/vmware2 type btrfs (rw,relatime,compress=lzo,space_cache)
/dev/md0 on /export type btrfs (rw,relatime,compress=lzo,space_cache)


# jornalctl -b d1964797ea13421eaf074d6be90790b6
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1815 of process 1671 (/usr/bi
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1812 of process 1671 (/usr/bi
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1809 of process 1671 (/usr/bi
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1671 of process 1671 (/usr/bi
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Demoted 8 threads.
 8月 20 21:25:31 myfedora kernel: general protection fault: 0000 [#1] SMP 
 8月 20 21:25:31 myfedora kernel: Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink arc4 md4 
 8月 20 21:25:31 myfedora kernel:  snd_hda_core joydev snd_compress snd_hwdep snd_pcm_dmaengine snd_se
 8月 20 21:25:31 myfedora kernel: CPU: 6 PID: 16443 Comm: kworker/u16:8 Tainted: G           OE   4.1.
 8月 20 21:25:31 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Ext
 8月 20 21:25:31 myfedora kernel: Workqueue: btrfs-submit btrfs_submit_helper [btrfs]
 8月 20 21:25:31 myfedora kernel: task: ffff88006b6889e0 ti: ffff880354270000 task.ti: ffff88035427000
 8月 20 21:25:31 myfedora kernel: RIP: 0010:[<ffffffffa0643630>]  [<ffffffffa0643630>] __find_stripe+0
 8月 20 21:25:31 myfedora kernel: RSP: 0018:ffff8803542739b8  EFLAGS: 00010006
 8月 20 21:25:31 myfedora kernel: RAX: 00008800c8da8000 RBX: 0000000000000000 RCX: ffff880354273a50
 8月 20 21:25:31 myfedora kernel: RDX: 00000000000004d8 RSI: 00000000bb1c54d8 RDI: ffff8807f843c000
 8月 20 21:25:31 myfedora kernel: RBP: ffff8803542739d8 R08: 0000000000000000 R09: 0000000000000000
 8月 20 21:25:31 myfedora kernel: R10: 0000000000000400 R11: 0000000000000003 R12: 00000000bb1c54d8
 8月 20 21:25:31 myfedora kernel: R13: ffff8807f843c000 R14: 00000000bb1c54d8 R15: 0000000000000000
 8月 20 21:25:31 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fb80000(0000) knlGS:00000000
 8月 20 21:25:31 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 20 21:25:31 myfedora kernel: CR2: 00007faefbc90000 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 20 21:25:31 myfedora kernel: Stack:
 8月 20 21:25:31 myfedora kernel:  0000000000000000 ffff8807f843c000 ffff8807f843c014 ffff8807f843c218
 8月 20 21:25:31 myfedora kernel:  ffff880354273a98 ffffffffa06471a7 ffff8807f272aa98 0000000000000246
 8月 20 21:25:31 myfedora kernel:  ffff880354273a48 ffffffff810e4c57 ffff880700000000 0000000300000283
 8月 20 21:25:31 myfedora kernel: Call Trace:
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa06471a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa064bca9>] make_request+0x1a9/0xca0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d8e00>] ? clear_state_bit+0x50/0x1a0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810d3d68>] ? __enqueue_entity+0x78/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e1ef2>] run_scheduled_bios+0x2a2/0x5b0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e2215>] pending_bios_fn+0x15/0x20 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03ed383>] btrfs_scrubnc_helper+0xc3/0x280 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03ed63e>] btrfs_submit_helper+0xe/0x10 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel: Code: 55 48 89 e5 41 55 41 54 53 49 89 fd 49 89 f4 89 d3 48 83 ec 08
 8月 20 21:25:31 myfedora kernel: RIP  [<ffffffffa0643630>] __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel:  RSP <ffff8803542739b8>
 8月 20 21:25:31 myfedora kernel: ---[ end trace 715386e437f2918d ]---
 8月 20 21:25:31 myfedora kernel: general protection fault: 0000 [#2] SMP 
 8月 20 21:25:31 myfedora kernel: Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink arc4 md4 
 8月 20 21:25:31 myfedora kernel:  snd_hda_core joydev snd_compress snd_hwdep snd_pcm_dmaengine snd_se
 8月 20 21:25:31 myfedora kernel: CPU: 7 PID: 17534 Comm: kworker/u16:18 Tainted: G      D    OE   4.1
 8月 20 21:25:31 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Ext
 8月 20 21:25:31 myfedora kernel: Workqueue: writeback bdi_writeback_workfn (flush-btrfs-3)
 8月 20 21:25:31 myfedora kernel: task: ffff880752bb3b40 ti: ffff88072c8e8000 task.ti: ffff88072c8e800
 8月 20 21:25:31 myfedora kernel: RIP: 0010:[<ffffffffa0643630>]  [<ffffffffa0643630>] __find_stripe+0
 8月 20 21:25:31 myfedora kernel: RSP: 0018:ffff88072c8eb498  EFLAGS: 00010006
 8月 20 21:25:31 myfedora kernel: RAX: 00008800c8da8000 RBX: 0000000000000000 RCX: ffff88072c8eb530
 8月 20 21:25:31 myfedora kernel: RDX: 00000000000002a0 RSI: 000000003d0bb2a0 RDI: ffff8807f843c000
 8月 20 21:25:31 myfedora kernel: RBP: ffff88072c8eb4b8 R08: 0000000000000000 R09: 0000000000000000
 8月 20 21:25:31 myfedora kernel: R10: 0000000000000400 R11: 0000000000000002 R12: 000000003d0bb2a0
 8月 20 21:25:31 myfedora kernel: R13: ffff8807f843c000 R14: 000000003d0bb2a0 R15: 0000000000000000
 8月 20 21:25:31 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fbc0000(0000) knlGS:00000000
 8月 20 21:25:31 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 20 21:25:31 myfedora kernel: CR2: 00007f66bf99b000 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 20 21:25:31 myfedora kernel: Stack:
 8月 20 21:25:31 myfedora kernel:  0000000000000000 ffff8807f843c000 ffff8807f843c018 ffff8807f843c218
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb578 ffffffffa06471a7 ffff8807f272aa98 0000000000000246
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb528 ffffffff810e4c57 ffff880700000000 0000000400000283
 8月 20 21:25:31 myfedora kernel: Call Trace:
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa06471a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa064bca9>] make_request+0x1a9/0xca0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813777ea>] ? bio_clone_bioset+0x8a/0x320
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e2713>] submit_stripe_bio+0x63/0x90 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e8ca6>] btrfs_map_bio+0x2f6/0x520 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b3879>] btree_submit_bio_hook+0xf9/0x100 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813764b9>] ? __bio_add_page.part.16+0x1d9/0x270
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d78f8>] submit_one_bio+0x68/0xa0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc587>] submit_extent_page.isra.36+0xd7/0x210 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc81c>] write_one_eb.isra.37+0x15c/0x200 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d7960>] ? end_extent_buffer_writeback+0x30/0x30 [btrfs
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03df25e>] btree_write_cache_pages+0x34e/0x480 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b2547>] btree_writepages+0x67/0x70 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811ba1fe>] do_writepages+0x1e/0x30
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81256b95>] __writeback_single_inode+0x45/0x2c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812571db>] writeback_sb_inodes+0x1db/0x4c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81257562>] __writeback_inodes_wb+0xa2/0xd0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812577e3>] wb_writeback+0x253/0x2e0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811b8fa0>] ? bdi_dirty_limit+0x40/0xe0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8125a1ec>] bdi_writeback_workfn+0x1dc/0x470
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810136d6>] ? __switch_to+0x216/0x5d0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel: Code: 55 48 89 e5 41 55 41 54 53 49 89 fd 49 89 f4 89 d3 48 83 ec 08
 8月 20 21:25:31 myfedora kernel: RIP  [<ffffffffa0643630>] __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel:  RSP <ffff88072c8eb498>
 8月 20 21:25:31 myfedora kernel: ---[ end trace 715386e437f2918e ]---
 8月 20 21:25:31 myfedora kernel: BUG: unable to handle kernel paging request at ffffffffffffffd8
 8月 20 21:25:31 myfedora kernel: IP: [<ffffffff810c1070>] kthread_data+0x10/0x20
 8月 20 21:25:31 myfedora kernel: PGD 1c0e067 PUD 1c10067 PMD 0 
 8月 20 21:25:31 myfedora kernel: Oops: 0000 [#3] SMP 
 8月 20 21:25:31 myfedora kernel: Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink arc4 md4 
 8月 20 21:25:31 myfedora kernel:  snd_hda_core joydev snd_compress snd_hwdep snd_pcm_dmaengine snd_se
 8月 20 21:25:31 myfedora kernel: CPU: 7 PID: 17534 Comm: kworker/u16:18 Tainted: G      D    OE   4.1
 8月 20 21:25:31 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Ext
 8月 20 21:25:31 myfedora kernel: task: ffff880752bb3b40 ti: ffff88072c8e8000 task.ti: ffff88072c8e800
 8月 20 21:25:31 myfedora kernel: RIP: 0010:[<ffffffff810c1070>]  [<ffffffff810c1070>] kthread_data+0x
 8月 20 21:25:31 myfedora kernel: RSP: 0018:ffff88072c8eb238  EFLAGS: 00010096
 8月 20 21:25:31 myfedora kernel: RAX: 0000000000000000 RBX: 0000000000000007 RCX: 0000000000000001
 8月 20 21:25:31 myfedora kernel: RDX: 0000000000000001 RSI: 0000000000000007 RDI: ffff880752bb3b40
 8月 20 21:25:31 myfedora kernel: RBP: ffff88072c8eb238 R08: ffff880752bb3bd0 R09: 0000000000000000
 8月 20 21:25:31 myfedora kernel: R10: 000000000000001d R11: 0000000000000000 R12: 0000000000017800
 8月 20 21:25:31 myfedora kernel: R13: ffff880752bb3b40 R14: 0000000000000007 R15: 0000000000000000
 8月 20 21:25:31 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fbc0000(0000) knlGS:00000000
 8月 20 21:25:31 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 20 21:25:31 myfedora kernel: CR2: 0000000000000028 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 20 21:25:31 myfedora kernel: Stack:
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb258 ffffffff810bb855 ffff88072c8eb258 ffff88081fbd7800
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb2a8 ffffffff8179dbe6 ffff88005609da90 ffff880752bb3b40
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb2a8 ffff88072c8ec000 ffff880752bb4250 ffff88072c8eb300
 8月 20 21:25:31 myfedora kernel: Call Trace:
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bb855>] wq_worker_sleeping+0x15/0xa0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8179dbe6>] __schedule+0x4d6/0x720
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8179de67>] schedule+0x37/0x90
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810a43e2>] do_exit+0x852/0xb30
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8101879f>] oops_end+0x9f/0xe0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81018c9b>] die+0x4b/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81015610>] do_general_protection+0x120/0x1b0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a3df8>] general_protection+0x28/0x30
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa0643630>] ? __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa06471a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa064bca9>] make_request+0x1a9/0xca0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813777ea>] ? bio_clone_bioset+0x8a/0x320
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e2713>] submit_stripe_bio+0x63/0x90 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e8ca6>] btrfs_map_bio+0x2f6/0x520 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b3879>] btree_submit_bio_hook+0xf9/0x100 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813764b9>] ? __bio_add_page.part.16+0x1d9/0x270
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d78f8>] submit_one_bio+0x68/0xa0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc587>] submit_extent_page.isra.36+0xd7/0x210 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc81c>] write_one_eb.isra.37+0x15c/0x200 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d7960>] ? end_extent_buffer_writeback+0x30/0x30 [btrfs
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03df25e>] btree_write_cache_pages+0x34e/0x480 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b2547>] btree_writepages+0x67/0x70 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811ba1fe>] do_writepages+0x1e/0x30
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81256b95>] __writeback_single_inode+0x45/0x2c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812571db>] writeback_sb_inodes+0x1db/0x4c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81257562>] __writeback_inodes_wb+0xa2/0xd0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812577e3>] wb_writeback+0x253/0x2e0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811b8fa0>] ? bdi_dirty_limit+0x40/0xe0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8125a1ec>] bdi_writeback_workfn+0x1dc/0x470
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810136d6>] ? __switch_to+0x216/0x5d0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel: Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f
 8月 20 21:25:31 myfedora kernel: RIP  [<ffffffff810c1070>] kthread_data+0x10/0x20
Comment 1 Yasuo Ohgaki 2015-08-20 17:03:38 EDT
Possible relevant event and info.

I was almost destroyed the RAID5 because I forgot to wipe filesystem by wipefs when I tried to replace a disk. I replaced faulty device (It's not broken, but had large latency) by 

mdadm /dev/md0 -r /dev/hdb -a /dev/hde
(I don't remember well if I used "-f /dev/sdb" option. I guess I did as man page of mdadm suggests.)

The RAID device started assembling disk and finished successfully. However, I forgot to remove old disk(/dev/sdb) and /dev/hde had GPT partition. Therefore, kernel detects /dev/sdb as outstanding disk and /dev/hde as non RAID device (recognized as usual GPT partitioned disk) on next reboot. /dev/sdb was active and clean state, but contained obsolete data. Kernel prevented activating the RAID thankfully.

I copied the RAID5 device's HDDs to other HDDs and attached copied HDDs to other PC so that data will not be lost. I assembled the copied RAID disks on another PC running CentOS7 and checked btrfs by btrfsck. There was no error.

Then I did the same on this PC. RAID assembled OK and btrfsck didn't report any errors. I suppose RAID and btrfs is not damaged at all, but it may be damaged by this.

Another change that may be relevant is kernel boot option. This PC has shutdown problem. It reboots rather than power off PC by shutdown. I ignored this because this PC is turned on always, but I added "acpi=force" option to see if it helps. It didn't help, but I left the option. I changed a BIOS option to see if it helps. I'll add this if this info is needed.

These were done just before the kernel panic.

I'm running Backintime to see if the same crash occurs. So far, it does not.

The PC's motherboard is ASRock Z97 Extreme4 with latest stable firmware. Since ASMedia SATA won't work with Linux, no HDD/device is attached to it. I'm using this PC about an year without kernel panic except I got panics when I tried to use "bcache + mdraid5 + btrfs" a year ago. 

"bcache + mdraid5 + btrfs" crashed consistently while copying few TB of data and broke bcache device (and related data in hdd) completely. I couldn't recover data in disks at all after the panic. I tried to copy few TB data 3 times and it happened 3 times, so I gave up bcache.


$ lspci
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0)
00:1c.5 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 6 (rev d0)
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
00:1c.7 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 8 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9120 SATA 6Gb/s Controller (rev 11)
06:00.1 IDE interface: Marvell Technology Group Ltd. 88SE912x IDE Controller (rev 11)

$ lsusb
Bus 006 Device 002: ID 8087:8001 Intel Corp. 
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID 0424:2504 Standard Microsystems Corp. USB 2.0 Hub
Bus 003 Device 004: ID 10d5:5532 Uni Class Technology Co., Ltd 
Bus 003 Device 002: ID 04cc:1520 ST-Ericsson USB 2.0 Hub (Avocent KVM)
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 002: ID 8087:8009 Intel Corp. 
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Comment 2 Yasuo Ohgaki 2015-08-21 21:46:48 EDT
Reproduced with kernel-4.1.5-200.fc22.x86_64 while running backintime, but it's not easy to reproduce. I'll try 4.1.4-200.fc22.x86_64 see if it reproduces this crash.

 8月 20 21:25:23 myfedora rtkit-daemon[910]: The canary thread is apparently starving. Taking action.
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Demoting known real-time threads.
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 2921 of process 2907 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 2918 of process 2907 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 2914 of process 2907 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 2907 of process 2907 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1815 of process 1671 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1812 of process 1671 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1809 of process 1671 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Successfully demoted thread 1671 of process 1671 (/usr/bin/pulseaudio).
 8月 20 21:25:23 myfedora rtkit-daemon[910]: Demoted 8 threads.
 8月 20 21:25:31 myfedora kernel: general protection fault: 0000 [#1] SMP 
 8月 20 21:25:31 myfedora kernel: Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink arc4 md4 nls_utf8 cifs dns_resolver fscache bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic snd_hda_codec_hdmi kvm snd_hda_intel snd_hda_controller iTCO_wdt crct10dif_pclmul snd_soc_rt5640 snd_hda_codec crc32_pclmul iTCO_vendor_support mxm_wmi snd_soc_rl6231 snd_soc_core ghash_clmulni_intel
 8月 20 21:25:31 myfedora kernel:  snd_hda_core joydev snd_compress snd_hwdep snd_pcm_dmaengine snd_seq i2c_i801 snd_seq_device lpc_ich snd_pcm tpm_tis mfd_core shpchp tpm snd_timer snd regmap_i2c mei_me dw_dmac dw_dmac_core soundcore i2c_designware_platform mei wmi snd_soc_sst_acpi i2c_designware_core acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel drm serio_raw e1000e pata_marvell uas usb_storage ptp pps_core sdhci_acpi sdhci mmc_core video i2c_hid
 8月 20 21:25:31 myfedora kernel: CPU: 6 PID: 16443 Comm: kworker/u16:8 Tainted: G           OE   4.1.5-200.fc22.x86_64 #1
 8月 20 21:25:31 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 20 21:25:31 myfedora kernel: Workqueue: btrfs-submit btrfs_submit_helper [btrfs]
 8月 20 21:25:31 myfedora kernel: task: ffff88006b6889e0 ti: ffff880354270000 task.ti: ffff880354270000
 8月 20 21:25:31 myfedora kernel: RIP: 0010:[<ffffffffa0643630>]  [<ffffffffa0643630>] __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel: RSP: 0018:ffff8803542739b8  EFLAGS: 00010006
 8月 20 21:25:31 myfedora kernel: RAX: 00008800c8da8000 RBX: 0000000000000000 RCX: ffff880354273a50
 8月 20 21:25:31 myfedora kernel: RDX: 00000000000004d8 RSI: 00000000bb1c54d8 RDI: ffff8807f843c000
 8月 20 21:25:31 myfedora kernel: RBP: ffff8803542739d8 R08: 0000000000000000 R09: 0000000000000000
 8月 20 21:25:31 myfedora kernel: R10: 0000000000000400 R11: 0000000000000003 R12: 00000000bb1c54d8
 8月 20 21:25:31 myfedora kernel: R13: ffff8807f843c000 R14: 00000000bb1c54d8 R15: 0000000000000000
 8月 20 21:25:31 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fb80000(0000) knlGS:0000000000000000
 8月 20 21:25:31 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 20 21:25:31 myfedora kernel: CR2: 00007faefbc90000 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 20 21:25:31 myfedora kernel: Stack:
 8月 20 21:25:31 myfedora kernel:  0000000000000000 ffff8807f843c000 ffff8807f843c014 ffff8807f843c218
 8月 20 21:25:31 myfedora kernel:  ffff880354273a98 ffffffffa06471a7 ffff8807f272aa98 0000000000000246
 8月 20 21:25:31 myfedora kernel:  ffff880354273a48 ffffffff810e4c57 ffff880700000000 0000000300000283
 8月 20 21:25:31 myfedora kernel: Call Trace:
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa06471a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa064bca9>] make_request+0x1a9/0xca0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d8e00>] ? clear_state_bit+0x50/0x1a0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810d3d68>] ? __enqueue_entity+0x78/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e1ef2>] run_scheduled_bios+0x2a2/0x5b0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e2215>] pending_bios_fn+0x15/0x20 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03ed383>] btrfs_scrubnc_helper+0xc3/0x280 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03ed63e>] btrfs_submit_helper+0xe/0x10 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel: Code: 55 48 89 e5 41 55 41 54 53 49 89 fd 49 89 f4 89 d3 48 83 ec 08 f6 05 99 16 01 00 04 75 5e 49 8b 45 00 4c 89 e2 81 e2 f8 0f 00 00 <48> 8b 04 02 48 85 c0 75 2d 31 db f6 05 50 16 01 00 04 75 57 48 
 8月 20 21:25:31 myfedora kernel: RIP  [<ffffffffa0643630>] __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel:  RSP <ffff8803542739b8>
 8月 20 21:25:31 myfedora kernel: ---[ end trace 715386e437f2918d ]---
 8月 20 21:25:31 myfedora kernel: general protection fault: 0000 [#2] SMP 
 8月 20 21:25:31 myfedora kernel: Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink arc4 md4 nls_utf8 cifs dns_resolver fscache bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic snd_hda_codec_hdmi kvm snd_hda_intel snd_hda_controller iTCO_wdt crct10dif_pclmul snd_soc_rt5640 snd_hda_codec crc32_pclmul iTCO_vendor_support mxm_wmi snd_soc_rl6231 snd_soc_core ghash_clmulni_intel
 8月 20 21:25:31 myfedora kernel:  snd_hda_core joydev snd_compress snd_hwdep snd_pcm_dmaengine snd_seq i2c_i801 snd_seq_device lpc_ich snd_pcm tpm_tis mfd_core shpchp tpm snd_timer snd regmap_i2c mei_me dw_dmac dw_dmac_core soundcore i2c_designware_platform mei wmi snd_soc_sst_acpi i2c_designware_core acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel drm serio_raw e1000e pata_marvell uas usb_storage ptp pps_core sdhci_acpi sdhci mmc_core video i2c_hid
 8月 20 21:25:31 myfedora kernel: CPU: 7 PID: 17534 Comm: kworker/u16:18 Tainted: G      D    OE   4.1.5-200.fc22.x86_64 #1
 8月 20 21:25:31 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 20 21:25:31 myfedora kernel: Workqueue: writeback bdi_writeback_workfn (flush-btrfs-3)
 8月 20 21:25:31 myfedora kernel: task: ffff880752bb3b40 ti: ffff88072c8e8000 task.ti: ffff88072c8e8000
 8月 20 21:25:31 myfedora kernel: RIP: 0010:[<ffffffffa0643630>]  [<ffffffffa0643630>] __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel: RSP: 0018:ffff88072c8eb498  EFLAGS: 00010006
 8月 20 21:25:31 myfedora kernel: RAX: 00008800c8da8000 RBX: 0000000000000000 RCX: ffff88072c8eb530
 8月 20 21:25:31 myfedora kernel: RDX: 00000000000002a0 RSI: 000000003d0bb2a0 RDI: ffff8807f843c000
 8月 20 21:25:31 myfedora kernel: RBP: ffff88072c8eb4b8 R08: 0000000000000000 R09: 0000000000000000
 8月 20 21:25:31 myfedora kernel: R10: 0000000000000400 R11: 0000000000000002 R12: 000000003d0bb2a0
 8月 20 21:25:31 myfedora kernel: R13: ffff8807f843c000 R14: 000000003d0bb2a0 R15: 0000000000000000
 8月 20 21:25:31 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fbc0000(0000) knlGS:0000000000000000
 8月 20 21:25:31 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 20 21:25:31 myfedora kernel: CR2: 00007f66bf99b000 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 20 21:25:31 myfedora kernel: Stack:
 8月 20 21:25:31 myfedora kernel:  0000000000000000 ffff8807f843c000 ffff8807f843c018 ffff8807f843c218
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb578 ffffffffa06471a7 ffff8807f272aa98 0000000000000246
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb528 ffffffff810e4c57 ffff880700000000 0000000400000283
 8月 20 21:25:31 myfedora kernel: Call Trace:
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa06471a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa064bca9>] make_request+0x1a9/0xca0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813777ea>] ? bio_clone_bioset+0x8a/0x320
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e2713>] submit_stripe_bio+0x63/0x90 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e8ca6>] btrfs_map_bio+0x2f6/0x520 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b3879>] btree_submit_bio_hook+0xf9/0x100 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813764b9>] ? __bio_add_page.part.16+0x1d9/0x270
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d78f8>] submit_one_bio+0x68/0xa0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc587>] submit_extent_page.isra.36+0xd7/0x210 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc81c>] write_one_eb.isra.37+0x15c/0x200 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d7960>] ? end_extent_buffer_writeback+0x30/0x30 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03df25e>] btree_write_cache_pages+0x34e/0x480 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b2547>] btree_writepages+0x67/0x70 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811ba1fe>] do_writepages+0x1e/0x30
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81256b95>] __writeback_single_inode+0x45/0x2c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812571db>] writeback_sb_inodes+0x1db/0x4c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81257562>] __writeback_inodes_wb+0xa2/0xd0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812577e3>] wb_writeback+0x253/0x2e0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811b8fa0>] ? bdi_dirty_limit+0x40/0xe0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8125a1ec>] bdi_writeback_workfn+0x1dc/0x470
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810136d6>] ? __switch_to+0x216/0x5d0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel: Code: 55 48 89 e5 41 55 41 54 53 49 89 fd 49 89 f4 89 d3 48 83 ec 08 f6 05 99 16 01 00 04 75 5e 49 8b 45 00 4c 89 e2 81 e2 f8 0f 00 00 <48> 8b 04 02 48 85 c0 75 2d 31 db f6 05 50 16 01 00 04 75 57 48 
 8月 20 21:25:31 myfedora kernel: RIP  [<ffffffffa0643630>] __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel:  RSP <ffff88072c8eb498>
 8月 20 21:25:31 myfedora kernel: ---[ end trace 715386e437f2918e ]---
 8月 20 21:25:31 myfedora kernel: BUG: unable to handle kernel paging request at ffffffffffffffd8
 8月 20 21:25:31 myfedora kernel: IP: [<ffffffff810c1070>] kthread_data+0x10/0x20
 8月 20 21:25:31 myfedora kernel: PGD 1c0e067 PUD 1c10067 PMD 0 
 8月 20 21:25:31 myfedora kernel: Oops: 0000 [#3] SMP 
 8月 20 21:25:31 myfedora kernel: Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink arc4 md4 nls_utf8 cifs dns_resolver fscache bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic snd_hda_codec_hdmi kvm snd_hda_intel snd_hda_controller iTCO_wdt crct10dif_pclmul snd_soc_rt5640 snd_hda_codec crc32_pclmul iTCO_vendor_support mxm_wmi snd_soc_rl6231 snd_soc_core ghash_clmulni_intel
 8月 20 21:25:31 myfedora kernel:  snd_hda_core joydev snd_compress snd_hwdep snd_pcm_dmaengine snd_seq i2c_i801 snd_seq_device lpc_ich snd_pcm tpm_tis mfd_core shpchp tpm snd_timer snd regmap_i2c mei_me dw_dmac dw_dmac_core soundcore i2c_designware_platform mei wmi snd_soc_sst_acpi i2c_designware_core acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel drm serio_raw e1000e pata_marvell uas usb_storage ptp pps_core sdhci_acpi sdhci mmc_core video i2c_hid
 8月 20 21:25:31 myfedora kernel: CPU: 7 PID: 17534 Comm: kworker/u16:18 Tainted: G      D    OE   4.1.5-200.fc22.x86_64 #1
 8月 20 21:25:31 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 20 21:25:31 myfedora kernel: task: ffff880752bb3b40 ti: ffff88072c8e8000 task.ti: ffff88072c8e8000
 8月 20 21:25:31 myfedora kernel: RIP: 0010:[<ffffffff810c1070>]  [<ffffffff810c1070>] kthread_data+0x10/0x20
 8月 20 21:25:31 myfedora kernel: RSP: 0018:ffff88072c8eb238  EFLAGS: 00010096
 8月 20 21:25:31 myfedora kernel: RAX: 0000000000000000 RBX: 0000000000000007 RCX: 0000000000000001
 8月 20 21:25:31 myfedora kernel: RDX: 0000000000000001 RSI: 0000000000000007 RDI: ffff880752bb3b40
 8月 20 21:25:31 myfedora kernel: RBP: ffff88072c8eb238 R08: ffff880752bb3bd0 R09: 0000000000000000
 8月 20 21:25:31 myfedora kernel: R10: 000000000000001d R11: 0000000000000000 R12: 0000000000017800
 8月 20 21:25:31 myfedora kernel: R13: ffff880752bb3b40 R14: 0000000000000007 R15: 0000000000000000
 8月 20 21:25:31 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fbc0000(0000) knlGS:0000000000000000
 8月 20 21:25:31 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 20 21:25:31 myfedora kernel: CR2: 0000000000000028 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 20 21:25:31 myfedora kernel: Stack:
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb258 ffffffff810bb855 ffff88072c8eb258 ffff88081fbd7800
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb2a8 ffffffff8179dbe6 ffff88005609da90 ffff880752bb3b40
 8月 20 21:25:31 myfedora kernel:  ffff88072c8eb2a8 ffff88072c8ec000 ffff880752bb4250 ffff88072c8eb300
 8月 20 21:25:31 myfedora kernel: Call Trace:
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bb855>] wq_worker_sleeping+0x15/0xa0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8179dbe6>] __schedule+0x4d6/0x720
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8179de67>] schedule+0x37/0x90
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810a43e2>] do_exit+0x852/0xb30
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8101879f>] oops_end+0x9f/0xe0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81018c9b>] die+0x4b/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81015610>] do_general_protection+0x120/0x1b0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a3df8>] general_protection+0x28/0x30
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa0643630>] ? __find_stripe+0x30/0xc0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa06471a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa064bca9>] make_request+0x1a9/0xca0 [raid456]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813777ea>] ? bio_clone_bioset+0x8a/0x320
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e2713>] submit_stripe_bio+0x63/0x90 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03e8ca6>] btrfs_map_bio+0x2f6/0x520 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b3879>] btree_submit_bio_hook+0xf9/0x100 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff813764b9>] ? __bio_add_page.part.16+0x1d9/0x270
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d78f8>] submit_one_bio+0x68/0xa0 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc587>] submit_extent_page.isra.36+0xd7/0x210 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03dc81c>] write_one_eb.isra.37+0x15c/0x200 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03d7960>] ? end_extent_buffer_writeback+0x30/0x30 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03df25e>] btree_write_cache_pages+0x34e/0x480 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffffa03b2547>] btree_writepages+0x67/0x70 [btrfs]
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811ba1fe>] do_writepages+0x1e/0x30
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81256b95>] __writeback_single_inode+0x45/0x2c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812571db>] writeback_sb_inodes+0x1db/0x4c0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff81257562>] __writeback_inodes_wb+0xa2/0xd0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff812577e3>] wb_writeback+0x253/0x2e0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff811b8fa0>] ? bdi_dirty_limit+0x40/0xe0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff8125a1ec>] bdi_writeback_workfn+0x1dc/0x470
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810136d6>] ? __switch_to+0x216/0x5d0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 20 21:25:31 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 20 21:25:31 myfedora kernel: Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 90 04 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 
 8月 20 21:25:31 myfedora kernel: RIP  [<ffffffff810c1070>] kthread_data+0x10/0x20
Comment 3 Yasuo Ohgaki 2015-08-25 21:33:16 EDT
Another oops occurred while no disk intensive programs are running. space_cache was disabled by fstab option this time. e.g.

UUID=cd13c2f0-2f2f-4cae-b994-7424581db35f /usr/local              btrfs   subvol=local,compress=lzo,nospcae_cache    0 0

Kernel is 4.1.5-200.fc22.x86_64. 

With this crash, kernel failed to mount /usr/local on boot. /usr/local is one of btrfs subvolume on /dev/md0. There are other subvolumes.

BTW, I'm using Fedora 20 to 22 on this system about a year didn't get these oops until recently.

$ mount | grep md0
/dev/md0 on /export type btrfs (rw,relatime,compress=lzo,nospace_cache)
/dev/md0 on /usr/local type btrfs (rw,relatime,compress=lzo,nospace_cache)
/dev/md0 on /home/backup type btrfs (rw,relatime,compress=lzo,nospace_cache)
/dev/md0 on /home/yohgaki/vmware2 type btrfs (rw,relatime,compress=lzo,nospace_cache)
/dev/md0 on /home/yohgaki/vmware type btrfs (rw,relatime,compress=lzo,nospace_cache)
/dev/md0 on /home/yohgaki/workspace type btrfs (rw,relatime,compress=lzo,nospace_cache)
 
Error logged on boot:
 8月 26 09:43:50 myfedora mount[768]: mount: wrong fs type, bad option, bad superblock on /dev/md0,
 8月 26 09:43:50 myfedora mount[768]: missing codepage or helper program, or other error
 8月 26 09:43:50 myfedora mount[768]: In some cases useful info is found in syslog - try
 8月 26 09:43:50 myfedora mount[768]: dmesg | tail or so.
 8月 26 09:43:50 myfedora kernel: BTRFS: open_ctree failed
 8月 26 09:43:50 myfedora kernel: BTRFS info (device md0): disabling disk space caching
 8月 26 09:43:51 myfedora systemd[1]: Mounted /boot.
 8月 26 09:43:51 myfedora systemd[1]: Started udev Wait for Complete Device Initialization.
 8月 26 09:43:51 myfedora audit[1]: <audit-1130> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-udev-se
 8月 26 09:43:51 myfedora systemd[1]: usr-local.mount mount process exited, code=exited status=32
 8月 26 09:43:51 myfedora systemd[1]: Failed to mount /usr/local.
 8月 26 09:43:51 myfedora systemd[1]: Dependency failed for Local File Systems.
 8月 26 09:43:51 myfedora systemd[1]: Dependency failed for Relabel all filesystems, if necessary.
 8月 26 09:43:51 myfedora systemd[1]: Job fedora-autorelabel.service/start failed with result 'dependency'.
 8月 26 09:43:51 myfedora systemd[1]: Dependency failed for Mark the need to relabel after reboot.
 8月 26 09:43:51 myfedora systemd[1]: Job fedora-autorelabel-mark.service/start failed with result 'dependency'.
 8月 26 09:43:51 myfedora systemd[1]: Job local-fs.target/start failed with result 'dependency'.
 8月 26 09:43:51 myfedora systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
 8月 26 09:43:51 myfedora systemd[1]: Unit usr-local.mount entered failed state.
 8月 26 09:43:51 myfedora systemd[1]: Started Load/Save Screen Backlight Brightness of backlight:acpi_video0.

Other subvolume mounted w/o error. I checked /dev/md0 by btrfsck without --repair option and it reported invalid space_cache, but no other error. It seems manual mount and umount of /usr/local fixed the error somehow.


Crash log:
 8月 26 09:37:13 myfedora audit[27664]: <audit-1326> auid=1000 uid=1000 gid=1000 ses=1 pid=27664 comm="chrome" exe="/opt/google/chrome/chrome" sig=0 arch=c000003e syscall=273 compat=0 ip=0x7f77e8f5f4f4 code=0x50000
 8月 26 09:37:14 myfedora audit[27665]: <audit-1326> auid=1000 uid=1000 gid=1000 ses=1 pid=27665 comm="chrome" exe="/opt/google/chrome/chrome" sig=0 arch=c000003e syscall=273 compat=0 ip=0x7f77e8f5f4f4 code=0x50000
 8月 26 09:37:21 myfedora gnome-session[2388]: [5:2:0826/093721:ERROR:channel.cc(300)] RawChannel read error (connection broken)
 8月 26 09:37:48 myfedora kernel: general protection fault: 0000 [#1] SMP 
 8月 26 09:37:48 myfedora kernel: Modules linked in: bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 intel_rapl async_raid6_recov iosf_mbi async_memcpy async_pq x86_pkg_temp_thermal async_xor async_tx iTCO_wdt iTCO_vendor_support coretemp snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi kvm_intel snd_hda_intel snd_soc_rt5640 snd_hda_controller kvm snd_soc_rl6231 snd_soc_core snd_hda_codec joydev snd_hda_core snd_compress crct10dif_pclmul snd_hwdep snd_pcm_dmaengine crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device
 8月 26 09:37:48 myfedora kernel:  snd_pcm i2c_i801 lpc_ich snd_timer mfd_core shpchp tpm_tis snd dw_dmac dw_dmac_core mei_me wmi regmap_i2c mei tpm soundcore i2c_designware_platform i2c_designware_core snd_soc_sst_acpi acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel e1000e serio_raw drm uas pata_marvell usb_storage sdhci_acpi sdhci ptp pps_core mmc_core video i2c_hid
 8月 26 09:37:48 myfedora kernel: CPU: 3 PID: 28879 Comm: kworker/u16:1 Tainted: G           OE   4.1.5-200.fc22.x86_64 #1
 8月 26 09:37:48 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 26 09:37:48 myfedora kernel: Workqueue: btrfs-submit btrfs_submit_helper [btrfs]
 8月 26 09:37:48 myfedora kernel: task: ffff88006f470000 ti: ffff8802cc8a4000 task.ti: ffff8802cc8a4000
 8月 26 09:37:48 myfedora kernel: RIP: 0010:[<ffffffffa0735630>]  [<ffffffffa0735630>] __find_stripe+0x30/0xc0 [raid456]
 8月 26 09:37:48 myfedora kernel: RSP: 0018:ffff8802cc8a79b8  EFLAGS: 00010006
 8月 26 09:37:48 myfedora kernel: RAX: 00008807fa848000 RBX: 0000000000000000 RCX: ffff8802cc8a7a50
 8月 26 09:37:48 myfedora kernel: RDX: 00000000000007e8 RSI: 00000000ae5957e8 RDI: ffff8807f0ecf800
 8月 26 09:37:48 myfedora kernel: RBP: ffff8802cc8a79d8 R08: 0000000000000000 R09: 0000000000000001
 8月 26 09:37:48 myfedora kernel: R10: 0000000000000400 R11: 0000000000000001 R12: 00000000ae5957e8
 8月 26 09:37:48 myfedora kernel: R13: ffff8807f0ecf800 R14: 00000000ae5957e8 R15: 0000000000000000
 8月 26 09:37:48 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fac0000(0000) knlGS:0000000000000000
 8月 26 09:37:48 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 26 09:37:48 myfedora kernel: CR2: 00007f92d6108000 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 26 09:37:48 myfedora kernel: Stack:
 8月 26 09:37:48 myfedora kernel:  0000000000000000 ffff8807f0ecf800 ffff8807f0ecf81c ffff8807f0ecfa18
 8月 26 09:37:48 myfedora kernel:  ffff8802cc8a7a98 ffffffffa07391a7 ffff8807f5ccba98 0000000000000246
 8月 26 09:37:48 myfedora kernel:  ffff8802cc8a7a48 ffffffff810e4c57 ffff880700000000 0000000500000283
 8月 26 09:37:48 myfedora kernel: Call Trace:
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa07391a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa073dca9>] make_request+0x1a9/0xca0 [raid456]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03cf500>] ? extent_writepages+0x60/0x90 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffff813d799a>] ? __percpu_counter_compare+0x2a/0x90
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 26 09:37:48 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 26 09:37:48 myfedora kernel:  [<ffffffff812572fb>] ? writeback_sb_inodes+0x2fb/0x4c0
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8110aed6>] ? mod_timer+0x136/0x230
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03d2ef2>] run_scheduled_bios+0x2a2/0x5b0 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03d3215>] pending_bios_fn+0x15/0x20 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03de383>] btrfs_scrubnc_helper+0xc3/0x280 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03de63e>] btrfs_submit_helper+0xe/0x10 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 26 09:37:48 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 26 09:37:48 myfedora kernel: Code: 55 48 89 e5 41 55 41 54 53 49 89 fd 49 89 f4 89 d3 48 83 ec 08 f6 05 99 16 01 00 04 75 5e 49 8b 45 00 4c 89 e2 81 e2 f8 0f 00 00 <48> 8b 04 02 48 85 c0 75 2d 31 db f6 05 50 16 01 00 04 75 57 48 
 8月 26 09:37:48 myfedora kernel: RIP  [<ffffffffa0735630>] __find_stripe+0x30/0xc0 [raid456]
 8月 26 09:37:48 myfedora kernel:  RSP <ffff8802cc8a79b8>
 8月 26 09:37:48 myfedora kernel: ---[ end trace 1a3369b84d382259 ]---
 8月 26 09:37:48 myfedora kernel: BUG: unable to handle kernel paging request at ffffffffffffffd8
 8月 26 09:37:48 myfedora kernel: IP: [<ffffffff810c1070>] kthread_data+0x10/0x20
 8月 26 09:37:48 myfedora kernel: PGD 1c0e067 PUD 1c10067 PMD 0 
 8月 26 09:37:48 myfedora kernel: Oops: 0000 [#2] SMP 
 8月 26 09:37:48 myfedora kernel: Modules linked in: bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 intel_rapl async_raid6_recov iosf_mbi async_memcpy async_pq x86_pkg_temp_thermal async_xor async_tx iTCO_wdt iTCO_vendor_support coretemp snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi kvm_intel snd_hda_intel snd_soc_rt5640 snd_hda_controller kvm snd_soc_rl6231 snd_soc_core snd_hda_codec joydev snd_hda_core snd_compress crct10dif_pclmul snd_hwdep snd_pcm_dmaengine crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device
 8月 26 09:37:48 myfedora kernel:  snd_pcm i2c_i801 lpc_ich snd_timer mfd_core shpchp tpm_tis snd dw_dmac dw_dmac_core mei_me wmi regmap_i2c mei tpm soundcore i2c_designware_platform i2c_designware_core snd_soc_sst_acpi acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel e1000e serio_raw drm uas pata_marvell usb_storage sdhci_acpi sdhci ptp pps_core mmc_core video i2c_hid
 8月 26 09:37:48 myfedora kernel: CPU: 3 PID: 28879 Comm: kworker/u16:1 Tainted: G      D    OE   4.1.5-200.fc22.x86_64 #1
 8月 26 09:37:48 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 26 09:37:48 myfedora kernel: task: ffff88006f470000 ti: ffff8802cc8a4000 task.ti: ffff8802cc8a4000
 8月 26 09:37:48 myfedora kernel: RIP: 0010:[<ffffffff810c1070>]  [<ffffffff810c1070>] kthread_data+0x10/0x20
 8月 26 09:37:48 myfedora kernel: RSP: 0018:ffff8802cc8a7758  EFLAGS: 00010092
 8月 26 09:37:48 myfedora kernel: RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000002
 8月 26 09:37:48 myfedora kernel: RDX: 0000000000000002 RSI: 0000000000000003 RDI: ffff88006f470000
 8月 26 09:37:48 myfedora kernel: RBP: ffff8802cc8a7758 R08: ffff88006f470090 R09: 0000000000000000
 8月 26 09:37:48 myfedora kernel: R10: 000000000000000a R11: 0000000000000000 R12: 0000000000017800
 8月 26 09:37:48 myfedora kernel: R13: ffff88006f470000 R14: 0000000000000003 R15: 0000000000000000
 8月 26 09:37:48 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fac0000(0000) knlGS:0000000000000000
 8月 26 09:37:48 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 26 09:37:48 myfedora kernel: CR2: 0000000000000028 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 26 09:37:48 myfedora kernel: Stack:
 8月 26 09:37:48 myfedora kernel:  ffff8802cc8a7778 ffffffff810bb855 ffff8802cc8a7778 ffff88081fad7800
 8月 26 09:37:48 myfedora kernel:  ffff8802cc8a77c8 ffffffff8179dbe6 ffff88078356c3a8 ffff88006f470000
 8月 26 09:37:48 myfedora kernel:  ffff8802cc8a77c8 ffff8802cc8a8000 ffff88006f470710 ffff8802cc8a7820
 8月 26 09:37:48 myfedora kernel: Call Trace:
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810bb855>] wq_worker_sleeping+0x15/0xa0
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8179dbe6>] __schedule+0x4d6/0x720
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8179de67>] schedule+0x37/0x90
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810a43e2>] do_exit+0x852/0xb30
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8101879f>] oops_end+0x9f/0xe0
 8月 26 09:37:48 myfedora kernel:  [<ffffffff81018c9b>] die+0x4b/0x70
 8月 26 09:37:48 myfedora kernel:  [<ffffffff81015610>] do_general_protection+0x120/0x1b0
 8月 26 09:37:48 myfedora kernel:  [<ffffffff817a3df8>] general_protection+0x28/0x30
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa0735630>] ? __find_stripe+0x30/0xc0 [raid456]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa07391a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4c57>] ? prepare_to_wait_event+0x87/0x100
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa073dca9>] make_request+0x1a9/0xca0 [raid456]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03cf500>] ? extent_writepages+0x60/0x90 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffff813d799a>] ? __percpu_counter_compare+0x2a/0x90
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 26 09:37:48 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 26 09:37:48 myfedora kernel:  [<ffffffff812572fb>] ? writeback_sb_inodes+0x2fb/0x4c0
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 26 09:37:48 myfedora kernel:  [<ffffffff8110aed6>] ? mod_timer+0x136/0x230
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03d2ef2>] run_scheduled_bios+0x2a2/0x5b0 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03d3215>] pending_bios_fn+0x15/0x20 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03de383>] btrfs_scrubnc_helper+0xc3/0x280 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffffa03de63e>] btrfs_submit_helper+0xe/0x10 [btrfs]
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810baa8b>] process_one_work+0x1bb/0x410
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810bad33>] worker_thread+0x53/0x480
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810bace0>] ? process_one_work+0x410/0x410
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 26 09:37:48 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 26 09:37:48 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 26 09:37:48 myfedora kernel: Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 90 04 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 
 8月 26 09:37:48 myfedora kernel: RIP  [<ffffffff810c1070>] kthread_data+0x10/0x20
 8月 26 09:37:48 myfedora kernel:  RSP <ffff8802cc8a7758>
 8月 26 09:37:48 myfedora kernel: CR2: ffffffffffffffd8
 8月 26 09:37:48 myfedora kernel: ---[ end trace 1a3369b84d38225a ]---
 8月 26 09:37:48 myfedora kernel: Fixing recursive fault but reboot is needed!
 8月 26 09:37:49 myfedora abrt-dump-journal-oops[2163]: abrt-dump-journal-oops: Found oopses: 2
 8月 26 09:37:49 myfedora abrt-dump-journal-oops[2163]: abrt-dump-journal-oops: Creating problem directories
 8月 26 09:37:51 myfedora rtkit-daemon[914]: The canary thread is apparently starving. Taking action.
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Demoting known real-time threads.
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 2950 of process 2905 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 2930 of process 2905 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 2912 of process 2905 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 2905 of process 2905 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 1606 of process 1472 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 1603 of process 1472 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 1516 of process 1472 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Successfully demoted thread 1472 of process 1472 (/usr/bin/pulseaudio).
 8月 26 09:37:51 myfedora rtkit-daemon[914]: Demoted 8 threads.
 8月 26 09:37:51 myfedora abrt-dump-journal-oops[2163]: Reported 2 kernel oopses to Abrt
 8月 26 09:37:54 myfedora kernel: general protection fault: 0000 [#3] SMP 
 8月 26 09:37:54 myfedora kernel: Modules linked in: bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 intel_rapl async_raid6_recov iosf_mbi async_memcpy async_pq x86_pkg_temp_thermal async_xor async_tx iTCO_wdt iTCO_vendor_support coretemp snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi kvm_intel snd_hda_intel snd_soc_rt5640 snd_hda_controller kvm snd_soc_rl6231 snd_soc_core snd_hda_codec joydev snd_hda_core snd_compress crct10dif_pclmul snd_hwdep snd_pcm_dmaengine crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device
 8月 26 09:37:54 myfedora kernel:  snd_pcm i2c_i801 lpc_ich snd_timer mfd_core shpchp tpm_tis snd dw_dmac dw_dmac_core mei_me wmi regmap_i2c mei tpm soundcore i2c_designware_platform i2c_designware_core snd_soc_sst_acpi acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel e1000e serio_raw drm uas pata_marvell usb_storage sdhci_acpi sdhci ptp pps_core mmc_core video i2c_hid
 8月 26 09:37:54 myfedora kernel: CPU: 7 PID: 871 Comm: btrfs-transacti Tainted: G      D    OE   4.1.5-200.fc22.x86_64 #1
 8月 26 09:37:54 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 26 09:37:54 myfedora kernel: task: ffff8807e6f81da0 ti: ffff8807e133c000 task.ti: ffff8807e133c000
 8月 26 09:37:54 myfedora kernel: RIP: 0010:[<ffffffffa0735630>]  [<ffffffffa0735630>] __find_stripe+0x30/0xc0 [raid456]
 8月 26 09:37:54 myfedora kernel: RSP: 0018:ffff8807e133f6a8  EFLAGS: 00010006
 8月 26 09:37:54 myfedora kernel: RAX: 00008807fa848000 RBX: 0000000000000000 RCX: ffff8807e133f740
 8月 26 09:37:54 myfedora kernel: RDX: 0000000000000660 RSI: 000000003fe17660 RDI: ffff8807f0ecf800
 8月 26 09:37:54 myfedora kernel: RBP: ffff8807e133f6c8 R08: 0000000000000000 R09: 0000000000000003
 8月 26 09:37:54 myfedora kernel: R10: 0000000000000400 R11: 0000000000000003 R12: 000000003fe17660
 8月 26 09:37:54 myfedora kernel: R13: ffff8807f0ecf800 R14: 000000003fe17660 R15: 0000000000000000
 8月 26 09:37:54 myfedora kernel: FS:  0000000000000000(0000) GS:ffff88081fbc0000(0000) knlGS:0000000000000000
 8月 26 09:37:54 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 26 09:37:54 myfedora kernel: CR2: 0000309f03e94000 CR3: 0000000001c0b000 CR4: 00000000001406e0
 8月 26 09:37:54 myfedora kernel: Stack:
 8月 26 09:37:54 myfedora kernel:  ffff8800c88bc9d0 ffff8807f0ecf800 ffff8807f0ecf818 ffff8807f0ecfa18
 8月 26 09:37:54 myfedora kernel:  ffff8807e133f788 ffffffffa07391a7 ffff8806427bf0b0 0000000000000002
 8月 26 09:37:54 myfedora kernel:  ffff8803c10081e0 ffff88065b84b518 ffff880700000000 00000004a03fa8a4
 8月 26 09:37:54 myfedora kernel: Call Trace:
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa07391a7>] get_active_stripe+0x137/0x710 [raid456]
 8月 26 09:37:54 myfedora kernel:  [<ffffffff810e4834>] ? prepare_to_wait+0x54/0x80
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa073dca9>] make_request+0x1a9/0xca0 [raid456]
 8月 26 09:37:54 myfedora kernel:  [<ffffffff810e4d40>] ? wake_atomic_t_function+0x70/0x70
 8月 26 09:37:54 myfedora kernel:  [<ffffffff815fda2c>] md_make_request+0xec/0x230
 8月 26 09:37:54 myfedora kernel:  [<ffffffff8137d300>] generic_make_request+0xe0/0x130
 8月 26 09:37:54 myfedora kernel:  [<ffffffff8137d3c8>] submit_bio+0x78/0x180
 8月 26 09:37:54 myfedora kernel:  [<ffffffff813777ea>] ? bio_clone_bioset+0x8a/0x320
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03d3713>] submit_stripe_bio+0x63/0x90 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03d9ca6>] btrfs_map_bio+0x2f6/0x520 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03a4879>] btree_submit_bio_hook+0xf9/0x100 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffff811bb500>] ? __pagevec_lru_add_fn+0x180/0x180
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03c8960>] ? end_extent_buffer_writeback+0x30/0x30 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03c88f8>] submit_one_bio+0x68/0xa0 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03c8ddb>] flush_epd_write_bio+0x3b/0x60 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03c8e0e>] flush_write_bio+0xe/0x10 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03d0198>] btree_write_cache_pages+0x288/0x480 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffff8120c015>] ? kmem_cache_alloc+0x1b5/0x240
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03cadb2>] ? convert_extent_bit+0x3d2/0x570 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03a3547>] btree_writepages+0x67/0x70 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffff811ba1fe>] do_writepages+0x1e/0x30
 8月 26 09:37:54 myfedora kernel:  [<ffffffff811add75>] __filemap_fdatawrite_range+0x65/0x90
 8月 26 09:37:54 myfedora kernel:  [<ffffffff811ade73>] filemap_fdatawrite_range+0x13/0x20
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03ab7c1>] btrfs_write_marked_extents+0x131/0x150 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa0428dbf>] btrfs_write_and_wait_marked_extents.constprop.22+0x40/0x7f [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03ac4b9>] btrfs_commit_transaction+0x879/0xab0 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03a7885>] transaction_kthread+0x245/0x260 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffffa03a7640>] ? btrfs_cleanup_transaction+0x560/0x560 [btrfs]
 8月 26 09:37:54 myfedora kernel:  [<ffffffff810c0ba8>] kthread+0xd8/0xf0
 8月 26 09:37:54 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 26 09:37:54 myfedora kernel:  [<ffffffff817a2262>] ret_from_fork+0x42/0x70
 8月 26 09:37:54 myfedora kernel:  [<ffffffff810c0ad0>] ? kthread_worker_fn+0x180/0x180
 8月 26 09:37:54 myfedora kernel: Code: 55 48 89 e5 41 55 41 54 53 49 89 fd 49 89 f4 89 d3 48 83 ec 08 f6 05 99 16 01 00 04 75 5e 49 8b 45 00 4c 89 e2 81 e2 f8 0f 00 00 <48> 8b 04 02 48 85 c0 75 2d 31 db f6 05 50 16 01 00 04 75 57 48 
 8月 26 09:37:54 myfedora kernel: RIP  [<ffffffffa0735630>] __find_stripe+0x30/0xc0 [raid456]
 8月 26 09:37:54 myfedora kernel:  RSP <ffff8807e133f6a8>
 8月 26 09:37:54 myfedora kernel: ---[ end trace 1a3369b84d38225b ]---
 8月 26 09:37:55 myfedora abrt-dump-journal-oops[2163]: abrt-dump-journal-oops: Found oopses: 1
 8月 26 09:37:55 myfedora abrt-dump-journal-oops[2163]: abrt-dump-journal-oops: Creating problem directories
 8月 26 09:38:00 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544910449324 new 18445303544909449077 attempts 1
 8月 26 09:38:01 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544909449077 new 18445303544908448634 attempts 1
 8月 26 09:38:02 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544908448634 new 18445303544907448281 attempts 1
 8月 26 09:38:03 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544907448281 new 18445303544906447895 attempts 1
 8月 26 09:38:04 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544906447895 new 18445303544905447570 attempts 1
 8月 26 09:38:05 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544905447570 new 18445303544904447221 attempts 1
 8月 26 09:38:06 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544904447221 new 18445303544903446888 attempts 1
 8月 26 09:38:07 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544903446888 new 18445303544902446548 attempts 1
 8月 26 09:38:08 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544902446548 new 18445303544901446218 attempts 1
 8月 26 09:38:09 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544901446218 new 18445303544900445720 attempts 1
 8月 26 09:38:10 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544900445720 new 18445303544899445336 attempts 1
 8月 26 09:38:11 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544899445336 new 18445303544898444923 attempts 1
 8月 26 09:38:12 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544898444923 new 18445303544897444501 attempts 1
 8月 26 09:38:13 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544897444501 new 18445303544896444064 attempts 1
 8月 26 09:38:14 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544896444064 new 18445303544895443020 attempts 1
 8月 26 09:38:15 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544895443020 new 18445303544894442669 attempts 1
 8月 26 09:38:16 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544894442669 new 18445303544893442303 attempts 1
 8月 26 09:38:17 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544893442303 new 18445303544892441975 attempts 1
 8月 26 09:38:18 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544892441975 new 18445303544891441616 attempts 1
 8月 26 09:38:19 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544891441616 new 18445303544890441264 attempts 1
 8月 26 09:38:20 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544890441264 new 18445303544889440884 attempts 1
 8月 26 09:38:20 myfedora kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [abrt-dump-journ:2163]
 8月 26 09:38:21 myfedora kernel: Modules linked in: bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 intel_rapl async_raid6_recov iosf_mbi async_memcpy async_pq x86_pkg_temp_thermal async_xor async_tx iTCO_wdt iTCO_vendor_support coretemp snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi kvm_intel snd_hda_intel snd_soc_rt5640 snd_hda_controller kvm snd_soc_rl6231 snd_soc_core snd_hda_codec joydev snd_hda_core snd_compress crct10dif_pclmul snd_hwdep snd_pcm_dmaengine crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device
 8月 26 09:38:21 myfedora kernel:  snd_pcm i2c_i801 lpc_ich snd_timer mfd_core shpchp tpm_tis snd dw_dmac dw_dmac_core mei_me wmi regmap_i2c mei tpm soundcore i2c_designware_platform i2c_designware_core snd_soc_sst_acpi acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel e1000e serio_raw drm uas pata_marvell usb_storage sdhci_acpi sdhci ptp pps_core mmc_core video i2c_hid
 8月 26 09:38:21 myfedora kernel: CPU: 0 PID: 2163 Comm: abrt-dump-journ Tainted: G      D    OE   4.1.5-200.fc22.x86_64 #1
 8月 26 09:38:21 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 26 09:38:21 myfedora kernel: task: ffff8807c8871da0 ti: ffff8807f91c8000 task.ti: ffff8807f91c8000
 8月 26 09:38:21 myfedora kernel: RIP: 0010:[<ffffffff81121cc5>]  [<ffffffff81121cc5>] smp_call_function_many+0x225/0x280
 8月 26 09:38:21 myfedora kernel: RSP: 0018:ffff8807f91cb698  EFLAGS: 00000202
 8月 26 09:38:21 myfedora kernel: RAX: 0000000000000003 RBX: 0000000000000293 RCX: 0000000000000001
 8月 26 09:38:21 myfedora kernel: RDX: ffff88081fa5b1f8 RSI: 0000000000000008 RDI: ffff8807ff406698
 8月 26 09:38:21 myfedora kernel: RBP: ffff8807f91cb6d8 R08: 0000000000000000 R09: 00000000000000fe
 8月 26 09:38:21 myfedora kernel: R10: ffffffff813ad019 R11: ffffffff813acfa4 R12: ffff8807f91cb668
 8月 26 09:38:21 myfedora kernel: R13: ffff8807f91cb628 R14: ffffffff813acfa4 R15: ffff8807ce58e2e8
 8月 26 09:38:21 myfedora kernel: FS:  00007f8fe5ecd840(0000) GS:ffff88081fa00000(0000) knlGS:0000000000000000
 8月 26 09:38:21 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 26 09:38:21 myfedora kernel: CR2: 00007fbc72d64000 CR3: 00000007dff57000 CR4: 00000000001406f0
 8月 26 09:38:21 myfedora kernel: Stack:
 8月 26 09:38:21 myfedora kernel:  0000000000000000 01ff880700000001 0000000000000003 0000000000000000
 8月 26 09:38:21 myfedora kernel:  ffffffff81fe5cc0 ffffffff811b4be0 0000000000000000 0000000000000007
 8月 26 09:38:21 myfedora kernel:  ffff8807f91cb708 ffffffff81121e08 0000000000000008 0000000000000000
 8月 26 09:38:21 myfedora kernel: Call Trace:
 8月 26 09:38:21 myfedora kernel:  [<ffffffff811b4be0>] ? page_alloc_cpu_notify+0x50/0x50
 8月 26 09:38:21 myfedora kernel:  [<ffffffff81121e08>] on_each_cpu_mask+0x28/0x60
 8月 26 09:38:21 myfedora kernel:  [<ffffffff811b29eb>] drain_all_pages+0xeb/0x100
 8月 26 09:38:21 myfedora kernel:  [<ffffffff811b6e20>] __alloc_pages_nodemask+0x650/0xa00
 8月 26 09:38:21 myfedora kernel:  [<ffffffff812003c1>] alloc_pages_current+0x91/0x110
 8月 26 09:38:21 myfedora kernel:  [<ffffffff81209725>] new_slab+0x85/0x4d0
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8120abca>] __slab_alloc+0x24a/0x5a0
 8月 26 09:38:21 myfedora kernel:  [<ffffffffa03bcda3>] ? btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:38:21 myfedora kernel:  [<ffffffff813c2815>] ? find_next_bit+0x15/0x30
 8月 26 09:38:21 myfedora kernel:  [<ffffffffa03bcda3>] ? btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8120c033>] kmem_cache_alloc+0x1d3/0x240
 8月 26 09:38:21 myfedora kernel:  [<ffffffffa03bcda3>] btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8124700d>] alloc_inode+0x1d/0xa0
 8月 26 09:38:21 myfedora kernel:  [<ffffffff81249063>] new_inode_pseudo+0x13/0x60
 8月 26 09:38:21 myfedora kernel:  [<ffffffff812490cb>] new_inode+0x1b/0x40
 8月 26 09:38:21 myfedora kernel:  [<ffffffffa03b9deb>] btrfs_new_inode+0x9b/0x5b0 [btrfs]
 8月 26 09:38:21 myfedora kernel:  [<ffffffffa03a2523>] ? btrfs_find_free_ino+0x133/0x140 [btrfs]
 8月 26 09:38:21 myfedora kernel:  [<ffffffffa03bb83f>] btrfs_create+0xff/0x220 [btrfs]
 8月 26 09:38:21 myfedora kernel:  [<ffffffff81237165>] vfs_create+0xd5/0x140
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8123a3c1>] do_last+0xbd1/0x1330
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8123ba3f>] path_openat+0x8f/0x680
 8月 26 09:38:21 myfedora kernel:  [<ffffffff81245177>] ? d_splice_alias+0xa7/0x2d0
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8123d689>] do_filp_open+0x49/0xd0
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8120c015>] ? kmem_cache_alloc+0x1b5/0x240
 8月 26 09:38:21 myfedora kernel:  [<ffffffff813c284a>] ? find_next_zero_bit+0x1a/0x30
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8124a4be>] ? __alloc_fd+0x7e/0x120
 8月 26 09:38:21 myfedora kernel:  [<ffffffff81229fda>] do_sys_open+0x13a/0x250
 8月 26 09:38:21 myfedora kernel:  [<ffffffff811473a6>] ? __audit_syscall_exit+0x1f6/0x290
 8月 26 09:38:21 myfedora kernel:  [<ffffffff8122a124>] SyS_openat+0x14/0x20
 8月 26 09:38:21 myfedora kernel:  [<ffffffff817a1e6e>] system_call_fastpath+0x12/0x71
 8月 26 09:38:21 myfedora kernel: Code: c0 00 89 c1 0f 8d 73 fe ff ff 48 98 49 8b 16 48 03 14 c5 a0 77 d2 81 8b 42 18 a8 01 74 c8 0f 1f 84 00 00 00 00 00 f3 90 8b 42 18 <a8> 01 75 f7 eb b5 0f b6 4d c8 4c 89 ea 4c 89 e6 44 89 f7 e8 83 
 8月 26 09:38:21 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544889440884 new 18445303544888440825 attempts 1
 8月 26 09:38:22 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544888440825 new 18445303544887440392 attempts 1
 8月 26 09:38:23 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544887440392 new 18445303544886440003 attempts 1
 8月 26 09:38:24 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544886440003 new 18445303544885439648 attempts 1
 8月 26 09:38:25 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544885439648 new 18445303544884439269 attempts 1
 8月 26 09:38:26 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544884439269 new 18445303544883438918 attempts 1
 8月 26 09:38:27 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544883438918 new 18445303544882438559 attempts 1
 8月 26 09:38:28 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544882438559 new 18445303544881438212 attempts 1
 8月 26 09:38:29 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544881438212 new 18445303544880437865 attempts 1
 8月 26 09:38:30 myfedora systemd[1]: systemd-logind.service watchdog timeout (limit 1min)!
 8月 26 09:38:30 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544880437865 new 18445303544879437437 attempts 1
 8月 26 09:38:31 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544879437437 new 18445303544878437073 attempts 1
 8月 26 09:38:32 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544878437073 new 18445303544877436728 attempts 1
 8月 26 09:38:33 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544877436728 new 18445303544876436276 attempts 1
 8月 26 09:38:34 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544876436276 new 18445303544875435879 attempts 1
 8月 26 09:38:35 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544875435879 new 18445303544874435508 attempts 1
 8月 26 09:38:36 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544874435508 new 18445303544873435145 attempts 1
 8月 26 09:38:37 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544873435145 new 18445303544872434759 attempts 1
 8月 26 09:38:38 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544872434759 new 18445303544871434408 attempts 1
 8月 26 09:38:39 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544871434408 new 18445303544870434048 attempts 1
 8月 26 09:38:40 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544870434048 new 18445303544869433674 attempts 1
 8月 26 09:38:41 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544869433674 new 18445303544868433314 attempts 1
 8月 26 09:38:42 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544868433314 new 18445303544867432954 attempts 1
 8月 26 09:38:43 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544867432954 new 18445303544866432497 attempts 1
 8月 26 09:38:44 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544866432497 new 18445303544865432084 attempts 1
 8月 26 09:38:45 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544865432084 new 18445303544864431697 attempts 1
 8月 26 09:38:46 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544864431697 new 18445303544863431342 attempts 1
 8月 26 09:38:47 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544863431342 new 18445303544862430931 attempts 1
 8月 26 09:38:48 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544862430931 new 18445303544861430578 attempts 1
 8月 26 09:38:48 myfedora kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [abrt-dump-journ:2163]
 8月 26 09:38:49 myfedora kernel: Modules linked in: bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 intel_rapl async_raid6_recov iosf_mbi async_memcpy async_pq x86_pkg_temp_thermal async_xor async_tx iTCO_wdt iTCO_vendor_support coretemp snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi kvm_intel snd_hda_intel snd_soc_rt5640 snd_hda_controller kvm snd_soc_rl6231 snd_soc_core snd_hda_codec joydev snd_hda_core snd_compress crct10dif_pclmul snd_hwdep snd_pcm_dmaengine crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device
 8月 26 09:38:49 myfedora kernel:  snd_pcm i2c_i801 lpc_ich snd_timer mfd_core shpchp tpm_tis snd dw_dmac dw_dmac_core mei_me wmi regmap_i2c mei tpm soundcore i2c_designware_platform i2c_designware_core snd_soc_sst_acpi acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel e1000e serio_raw drm uas pata_marvell usb_storage sdhci_acpi sdhci ptp pps_core mmc_core video i2c_hid
 8月 26 09:38:49 myfedora kernel: CPU: 0 PID: 2163 Comm: abrt-dump-journ Tainted: G      D    OEL  4.1.5-200.fc22.x86_64 #1
 8月 26 09:38:49 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 26 09:38:49 myfedora kernel: task: ffff8807c8871da0 ti: ffff8807f91c8000 task.ti: ffff8807f91c8000
 8月 26 09:38:49 myfedora kernel: RIP: 0010:[<ffffffff81121cc5>]  [<ffffffff81121cc5>] smp_call_function_many+0x225/0x280
 8月 26 09:38:49 myfedora kernel: RSP: 0018:ffff8807f91cb698  EFLAGS: 00000202
 8月 26 09:38:49 myfedora kernel: RAX: 0000000000000003 RBX: 0000000000000293 RCX: 0000000000000001
 8月 26 09:38:49 myfedora kernel: RDX: ffff88081fa5b1f8 RSI: 0000000000000008 RDI: ffff8807ff406698
 8月 26 09:38:49 myfedora kernel: RBP: ffff8807f91cb6d8 R08: 0000000000000000 R09: 00000000000000fe
 8月 26 09:38:49 myfedora kernel: R10: ffffffff813ad019 R11: ffffffff813acfa4 R12: ffff8807f91cb668
 8月 26 09:38:49 myfedora kernel: R13: ffff8807f91cb628 R14: ffffffff813acfa4 R15: ffff8807ce58e2e8
 8月 26 09:38:49 myfedora kernel: FS:  00007f8fe5ecd840(0000) GS:ffff88081fa00000(0000) knlGS:0000000000000000
 8月 26 09:38:49 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 26 09:38:49 myfedora kernel: CR2: 00007fbc72d64000 CR3: 00000007dff57000 CR4: 00000000001406f0
 8月 26 09:38:49 myfedora kernel: Stack:
 8月 26 09:38:49 myfedora kernel:  0000000000000000 01ff880700000001 0000000000000003 0000000000000000
 8月 26 09:38:49 myfedora kernel:  ffffffff81fe5cc0 ffffffff811b4be0 0000000000000000 0000000000000007
 8月 26 09:38:49 myfedora kernel:  ffff8807f91cb708 ffffffff81121e08 0000000000000008 0000000000000000
 8月 26 09:38:49 myfedora kernel: Call Trace:
 8月 26 09:38:49 myfedora kernel:  [<ffffffff811b4be0>] ? page_alloc_cpu_notify+0x50/0x50
 8月 26 09:38:49 myfedora kernel:  [<ffffffff81121e08>] on_each_cpu_mask+0x28/0x60
 8月 26 09:38:49 myfedora kernel:  [<ffffffff811b29eb>] drain_all_pages+0xeb/0x100
 8月 26 09:38:49 myfedora kernel:  [<ffffffff811b6e20>] __alloc_pages_nodemask+0x650/0xa00
 8月 26 09:38:49 myfedora kernel:  [<ffffffff812003c1>] alloc_pages_current+0x91/0x110
 8月 26 09:38:49 myfedora kernel:  [<ffffffff81209725>] new_slab+0x85/0x4d0
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8120abca>] __slab_alloc+0x24a/0x5a0
 8月 26 09:38:49 myfedora kernel:  [<ffffffffa03bcda3>] ? btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:38:49 myfedora kernel:  [<ffffffff813c2815>] ? find_next_bit+0x15/0x30
 8月 26 09:38:49 myfedora kernel:  [<ffffffffa03bcda3>] ? btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8120c033>] kmem_cache_alloc+0x1d3/0x240
 8月 26 09:38:49 myfedora kernel:  [<ffffffffa03bcda3>] btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8124700d>] alloc_inode+0x1d/0xa0
 8月 26 09:38:49 myfedora kernel:  [<ffffffff81249063>] new_inode_pseudo+0x13/0x60
 8月 26 09:38:49 myfedora kernel:  [<ffffffff812490cb>] new_inode+0x1b/0x40
 8月 26 09:38:49 myfedora kernel:  [<ffffffffa03b9deb>] btrfs_new_inode+0x9b/0x5b0 [btrfs]
 8月 26 09:38:49 myfedora kernel:  [<ffffffffa03a2523>] ? btrfs_find_free_ino+0x133/0x140 [btrfs]
 8月 26 09:38:49 myfedora kernel:  [<ffffffffa03bb83f>] btrfs_create+0xff/0x220 [btrfs]
 8月 26 09:38:49 myfedora kernel:  [<ffffffff81237165>] vfs_create+0xd5/0x140
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8123a3c1>] do_last+0xbd1/0x1330
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8123ba3f>] path_openat+0x8f/0x680
 8月 26 09:38:49 myfedora kernel:  [<ffffffff81245177>] ? d_splice_alias+0xa7/0x2d0
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8123d689>] do_filp_open+0x49/0xd0
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8120c015>] ? kmem_cache_alloc+0x1b5/0x240
 8月 26 09:38:49 myfedora kernel:  [<ffffffff813c284a>] ? find_next_zero_bit+0x1a/0x30
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8124a4be>] ? __alloc_fd+0x7e/0x120
 8月 26 09:38:49 myfedora kernel:  [<ffffffff81229fda>] do_sys_open+0x13a/0x250
 8月 26 09:38:49 myfedora kernel:  [<ffffffff811473a6>] ? __audit_syscall_exit+0x1f6/0x290
 8月 26 09:38:49 myfedora kernel:  [<ffffffff8122a124>] SyS_openat+0x14/0x20
 8月 26 09:38:49 myfedora kernel:  [<ffffffff817a1e6e>] system_call_fastpath+0x12/0x71
 8月 26 09:38:49 myfedora kernel: Code: c0 00 89 c1 0f 8d 73 fe ff ff 48 98 49 8b 16 48 03 14 c5 a0 77 d2 81 8b 42 18 a8 01 74 c8 0f 1f 84 00 00 00 00 00 f3 90 8b 42 18 <a8> 01 75 f7 eb b5 0f b6 4d c8 4c 89 ea 4c 89 e6 44 89 f7 e8 83 
 8月 26 09:38:49 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544861430578 new 18445303544860429995 attempts 1
 8月 26 09:38:50 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544860429995 new 18445303544859429617 attempts 1
 8月 26 09:38:51 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544859429617 new 18445303544858429163 attempts 1
 8月 26 09:38:52 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544858429163 new 18445303544857428698 attempts 1
 8月 26 09:38:53 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544857428698 new 18445303544856428326 attempts 1
 8月 26 09:38:54 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544856428326 new 18445303544855427893 attempts 1
 8月 26 09:38:55 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544855427893 new 18445303544854427497 attempts 1
 8月 26 09:38:56 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544854427497 new 18445303544853427048 attempts 1
 8月 26 09:38:57 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544853427048 new 18445303544852426671 attempts 1
 8月 26 09:38:58 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544852426671 new 18445303544851426314 attempts 1
 8月 26 09:38:59 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544851426314 new 18445303544850425943 attempts 1
 8月 26 09:39:00 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544850425943 new 18445303544849425542 attempts 1
 8月 26 09:39:01 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544849425542 new 18445303544848425167 attempts 1
 8月 26 09:39:02 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544848425167 new 18445303544847424702 attempts 1
 8月 26 09:39:03 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544847424702 new 18445303544846424239 attempts 1
 8月 26 09:39:04 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544846424239 new 18445303544845423852 attempts 1
 8月 26 09:39:05 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544845423852 new 18445303544844423476 attempts 1
 8月 26 09:39:06 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544844423476 new 18445303544843423101 attempts 1
 8月 26 09:39:07 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544843423101 new 18445303544842422727 attempts 1
 8月 26 09:39:08 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544842422727 new 18445303544841422330 attempts 1
 8月 26 09:39:09 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544841422330 new 18445303544840421807 attempts 1
 8月 26 09:39:10 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544840421807 new 18445303544839421411 attempts 1
 8月 26 09:39:11 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544839421411 new 18445303544838420923 attempts 1
 8月 26 09:39:12 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544838420923 new 18445303544837420547 attempts 1
 8月 26 09:39:13 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544837420547 new 18445303544836420167 attempts 1
 8月 26 09:39:14 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544836420167 new 18445303544835419787 attempts 1
 8月 26 09:39:15 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544835419787 new 18445303544834419323 attempts 1
 8月 26 09:39:16 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544834419323 new 18445303544833418938 attempts 1
 8月 26 09:39:16 myfedora kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [abrt-dump-journ:2163]
 8月 26 09:39:17 myfedora kernel: Modules linked in: bnep bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables vmnet(OE) ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci vmmon(OE) raid456 intel_rapl async_raid6_recov iosf_mbi async_memcpy async_pq x86_pkg_temp_thermal async_xor async_tx iTCO_wdt iTCO_vendor_support coretemp snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi kvm_intel snd_hda_intel snd_soc_rt5640 snd_hda_controller kvm snd_soc_rl6231 snd_soc_core snd_hda_codec joydev snd_hda_core snd_compress crct10dif_pclmul snd_hwdep snd_pcm_dmaengine crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device
 8月 26 09:39:17 myfedora kernel:  snd_pcm i2c_i801 lpc_ich snd_timer mfd_core shpchp tpm_tis snd dw_dmac dw_dmac_core mei_me wmi regmap_i2c mei tpm soundcore i2c_designware_platform i2c_designware_core snd_soc_sst_acpi acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi 8021q garp stp llc mrp btrfs xor i915 raid6_pq i2c_algo_bit drm_kms_helper crc32c_intel e1000e serio_raw drm uas pata_marvell usb_storage sdhci_acpi sdhci ptp pps_core mmc_core video i2c_hid
 8月 26 09:39:17 myfedora kernel: CPU: 0 PID: 2163 Comm: abrt-dump-journ Tainted: G      D    OEL  4.1.5-200.fc22.x86_64 #1
 8月 26 09:39:17 myfedora kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 26 09:39:17 myfedora kernel: task: ffff8807c8871da0 ti: ffff8807f91c8000 task.ti: ffff8807f91c8000
 8月 26 09:39:17 myfedora kernel: RIP: 0010:[<ffffffff81121cc5>]  [<ffffffff81121cc5>] smp_call_function_many+0x225/0x280
 8月 26 09:39:17 myfedora kernel: RSP: 0018:ffff8807f91cb698  EFLAGS: 00000202
 8月 26 09:39:17 myfedora kernel: RAX: 0000000000000003 RBX: 0000000000000293 RCX: 0000000000000001
 8月 26 09:39:17 myfedora kernel: RDX: ffff88081fa5b1f8 RSI: 0000000000000008 RDI: ffff8807ff406698
 8月 26 09:39:17 myfedora kernel: RBP: ffff8807f91cb6d8 R08: 0000000000000000 R09: 00000000000000fe
 8月 26 09:39:17 myfedora kernel: R10: ffffffff813ad019 R11: ffffffff813acfa4 R12: ffff8807f91cb668
 8月 26 09:39:17 myfedora kernel: R13: ffff8807f91cb628 R14: ffffffff813acfa4 R15: ffff8807ce58e2e8
 8月 26 09:39:17 myfedora kernel: FS:  00007f8fe5ecd840(0000) GS:ffff88081fa00000(0000) knlGS:0000000000000000
 8月 26 09:39:17 myfedora kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 26 09:39:17 myfedora kernel: CR2: 00007fbc72d64000 CR3: 00000007dff57000 CR4: 00000000001406f0
 8月 26 09:39:17 myfedora kernel: Stack:
 8月 26 09:39:17 myfedora kernel:  0000000000000000 01ff880700000001 0000000000000003 0000000000000000
 8月 26 09:39:17 myfedora kernel:  ffffffff81fe5cc0 ffffffff811b4be0 0000000000000000 0000000000000007
 8月 26 09:39:17 myfedora kernel:  ffff8807f91cb708 ffffffff81121e08 0000000000000008 0000000000000000
 8月 26 09:39:17 myfedora kernel: Call Trace:
 8月 26 09:39:17 myfedora kernel:  [<ffffffff811b4be0>] ? page_alloc_cpu_notify+0x50/0x50
 8月 26 09:39:17 myfedora kernel:  [<ffffffff81121e08>] on_each_cpu_mask+0x28/0x60
 8月 26 09:39:17 myfedora kernel:  [<ffffffff811b29eb>] drain_all_pages+0xeb/0x100
 8月 26 09:39:17 myfedora kernel:  [<ffffffff811b6e20>] __alloc_pages_nodemask+0x650/0xa00
 8月 26 09:39:17 myfedora kernel:  [<ffffffff812003c1>] alloc_pages_current+0x91/0x110
 8月 26 09:39:17 myfedora kernel:  [<ffffffff81209725>] new_slab+0x85/0x4d0
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8120abca>] __slab_alloc+0x24a/0x5a0
 8月 26 09:39:17 myfedora kernel:  [<ffffffffa03bcda3>] ? btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:39:17 myfedora kernel:  [<ffffffff813c2815>] ? find_next_bit+0x15/0x30
 8月 26 09:39:17 myfedora kernel:  [<ffffffffa03bcda3>] ? btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8120c033>] kmem_cache_alloc+0x1d3/0x240
 8月 26 09:39:17 myfedora kernel:  [<ffffffffa03bcda3>] btrfs_alloc_inode+0x23/0x1f0 [btrfs]
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8124700d>] alloc_inode+0x1d/0xa0
 8月 26 09:39:17 myfedora kernel:  [<ffffffff81249063>] new_inode_pseudo+0x13/0x60
 8月 26 09:39:17 myfedora kernel:  [<ffffffff812490cb>] new_inode+0x1b/0x40
 8月 26 09:39:17 myfedora kernel:  [<ffffffffa03b9deb>] btrfs_new_inode+0x9b/0x5b0 [btrfs]
 8月 26 09:39:17 myfedora kernel:  [<ffffffffa03a2523>] ? btrfs_find_free_ino+0x133/0x140 [btrfs]
 8月 26 09:39:17 myfedora kernel:  [<ffffffffa03bb83f>] btrfs_create+0xff/0x220 [btrfs]
 8月 26 09:39:17 myfedora kernel:  [<ffffffff81237165>] vfs_create+0xd5/0x140
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8123a3c1>] do_last+0xbd1/0x1330
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8123ba3f>] path_openat+0x8f/0x680
 8月 26 09:39:17 myfedora kernel:  [<ffffffff81245177>] ? d_splice_alias+0xa7/0x2d0
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8123d689>] do_filp_open+0x49/0xd0
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8120c015>] ? kmem_cache_alloc+0x1b5/0x240
 8月 26 09:39:17 myfedora kernel:  [<ffffffff813c284a>] ? find_next_zero_bit+0x1a/0x30
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8124a4be>] ? __alloc_fd+0x7e/0x120
 8月 26 09:39:17 myfedora kernel:  [<ffffffff81229fda>] do_sys_open+0x13a/0x250
 8月 26 09:39:17 myfedora kernel:  [<ffffffff811473a6>] ? __audit_syscall_exit+0x1f6/0x290
 8月 26 09:39:17 myfedora kernel:  [<ffffffff8122a124>] SyS_openat+0x14/0x20
 8月 26 09:39:17 myfedora kernel:  [<ffffffff817a1e6e>] system_call_fastpath+0x12/0x71
 8月 26 09:39:17 myfedora kernel: Code: c0 00 89 c1 0f 8d 73 fe ff ff 48 98 49 8b 16 48 03 14 c5 a0 77 d2 81 8b 42 18 a8 01 74 c8 0f 1f 84 00 00 00 00 00 f3 90 8b 42 18 <a8> 01 75 f7 eb b5 0f b6 4d c8 4c 89 ea 4c 89 e6 44 89 f7 e8 83 
 8月 26 09:39:17 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544833418938 new 18445303544832418757 attempts 1
 8月 26 09:39:18 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544832418757 new 18445303544831418382 attempts 1
 8月 26 09:39:19 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544831418382 new 18445303544830417917 attempts 1
 8月 26 09:39:20 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544830417917 new 18445303544829417466 attempts 1
 8月 26 09:39:21 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544829417466 new 18445303544828417086 attempts 1
 8月 26 09:39:22 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544828417086 new 18445303544827416716 attempts 1
 8月 26 09:39:23 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544827416716 new 18445303544826416230 attempts 1
 8月 26 09:39:24 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544826416230 new 18445303544825415801 attempts 1
 8月 26 09:39:25 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544825415801 new 18445303544824415401 attempts 1
 8月 26 09:39:26 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544824415401 new 18445303544823415007 attempts 1
 8月 26 09:39:27 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544823415007 new 18445303544822414592 attempts 1
 8月 26 09:39:28 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544822414592 new 18445303544821414204 attempts 1
 8月 26 09:39:29 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544821414204 new 18445303544820413827 attempts 1
 8月 26 09:39:30 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544820413827 new 18445303544819413421 attempts 1
 8月 26 09:39:31 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544819413421 new 18445303544818413039 attempts 1
 8月 26 09:39:32 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544818413039 new 18445303544817412571 attempts 1
 8月 26 09:39:33 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544817412571 new 18445303544816412170 attempts 1
 8月 26 09:39:34 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544816412170 new 18445303544815411709 attempts 1
 8月 26 09:39:35 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544815411709 new 18445303544814411329 attempts 1
 8月 26 09:39:36 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544814411329 new 18445303544813410892 attempts 1
 8月 26 09:39:37 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544813410892 new 18445303544812410515 attempts 1
 8月 26 09:39:38 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544812410515 new 18445303544811410118 attempts 1
 8月 26 09:39:39 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544811410118 new 18445303544810409722 attempts 1
 8月 26 09:39:40 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544810409722 new 18445303544809409313 attempts 1
 8月 26 09:39:41 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544809409313 new 18445303544808408929 attempts 1
 8月 26 09:39:42 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544808408929 new 18445303544807408462 attempts 1
 8月 26 09:39:43 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544807408462 new 18445303544806408063 attempts 1
 8月 26 09:39:44 myfedora kernel: /dev/vmmon[7971]: HostIFReadUptimeWork: detected settimeofday: fixed uptimeBase old 18445303544806408063 new 18445303544805407682 attempts 1
 8月 26 09:39:44 myfedora kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [abrt-dump-journ:2163]
Comment 4 Yasuo Ohgaki 2015-08-26 07:10:06 EDT
Installed debug kernel and got this oops.
I also checked system by memtest86+ and it does not report error.

I'm using VMware Workstation 11.1.2 build-2780323 (latest 11 version)

 8月 26 19:59:58 dev.inter.es-i.jp kernel: ------------[ cut here ]------------
 8月 26 19:59:58 dev.inter.es-i.jp kernel: WARNING: CPU: 2 PID: 4281 at kernel/sched/core.c:7286 __might_sleep+0x87/0x90()
 8月 26 19:59:58 dev.inter.es-i.jp kernel: do not call blocking ops when !TASK_RUNNING; state=1 set at [<ffffffff811031ad>] prepare_to_wait+0x2d/0x90
 8月 26 19:59:58 dev.inter.es-i.jp kernel: Modules linked in: bnep bluetooth rfkill vmnet(OE) vmmon(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci iTCO_wdt iTCO_vendor_support mxm_wmi raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx intel_rapl iosf_mbi joydev x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_soc_rt5640 snd_soc_rl6231 snd_hda_intel snd_soc_core snd_hda_controller snd_compress snd_hda_codec snd_pcm_dmaengine snd_hda_core snd_hwdep snd_seq lpc_ich
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  i2c_i801 snd_seq_device mfd_core snd_pcm shpchp mei_me mei snd_timer snd tpm_tis soundcore regmap_i2c tpm dw_dmac i2c_designware_platform snd_soc_sst_acpi wmi dw_dmac_core i2c_designware_core acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi btrfs 8021q garp stp llc mrp xor raid6_pq crc32c_intel serio_raw i915 uas pata_marvell i2c_algo_bit usb_storage drm_kms_helper e1000e drm ptp pps_core sdhci_acpi sdhci mmc_core video i2c_hid
 8月 26 19:59:58 dev.inter.es-i.jp kernel: CPU: 2 PID: 4281 Comm: vmware Tainted: G           OE   4.1.5-200.fc22.x86_64+debug #1
 8月 26 19:59:58 dev.inter.es-i.jp kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  0000000000000000 00000000f09ebddc ffff880771f93b18 ffffffff81896885
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  0000000000000000 ffff880771f93b70 ffff880771f93b58 ffffffff810b14fa
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  0000000000000001 ffffffff81c6801d 0000000000000268 0000000000000000
 8月 26 19:59:58 dev.inter.es-i.jp kernel: Call Trace:
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff81896885>] dump_stack+0x4c/0x65
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff810b14fa>] warn_slowpath_common+0x8a/0xc0
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff810b1585>] warn_slowpath_fmt+0x55/0x70
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff811031ad>] ? prepare_to_wait+0x2d/0x90
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff811031ad>] ? prepare_to_wait+0x2d/0x90
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff810e19f7>] __might_sleep+0x87/0x90
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff8189c5dd>] mutex_lock_nested+0x3d/0x460
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff8111027f>] ? mark_held_locks+0x7f/0xc0
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff8189fc36>] ? _raw_spin_unlock_irqrestore+0x36/0x70
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff811103fd>] ? trace_hardirqs_on_caller+0x13d/0x1e0
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffffa0767616>] vmci_qpair_produce_free_space+0xd6/0x110 [vmw_vmci]
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffffa077ac51>] ? vsock_stream_sendmsg+0x71/0x330 [vsock]
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffffa07847d9>] vmci_transport_stream_has_space+0x19/0x20 [vmw_vsock_vmci_transport]
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffffa077ad6e>] vsock_stream_sendmsg+0x18e/0x330 [vsock]
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff81103720>] ? wake_atomic_t_function+0x70/0x70
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff8172a12d>] sock_sendmsg+0x3d/0x50
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff8172a1cc>] sock_write_iter+0x8c/0x100
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff812816b4>] __vfs_write+0xd4/0x110
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff812820a9>] vfs_write+0xa9/0x1c0
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff8102cd45>] ? do_audit_syscall_entry+0x55/0x80
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff81282e3c>] SyS_write+0x5c/0xd0
 8月 26 19:59:58 dev.inter.es-i.jp kernel:  [<ffffffff818a066e>] system_call_fastpath+0x12/0x76
 8月 26 19:59:58 dev.inter.es-i.jp kernel: ---[ end trace 14c1f19183245702 ]---
Comment 5 Yasuo Ohgaki 2015-08-26 17:19:36 EDT
(In reply to Yasuo Ohgaki from comment #3)
> Another oops occurred while no disk intensive programs are running.
> space_cache was disabled by fstab option this time. e.g.
> 
> UUID=cd13c2f0-2f2f-4cae-b994-7424581db35f /usr/local              btrfs  
> subvol=local,compress=lzo,nospcae_cache    0 0
> 
> Kernel is 4.1.5-200.fc22.x86_64. 
> 
> With this crash, kernel failed to mount /usr/local on boot. /usr/local is
> one of btrfs subvolume on /dev/md0. There are other subvolumes.

I tried 4.1.5 debug kernel and chrome/vmware were killed by OOM next morning and system hanged up according to log.

One of the reason why kernel start failing to mount /usr/local may be typo in /etc/fstab. I spelled fs option wrong, i.e. nospcae_cache rather than nospace_cache. Found this typo by "dmesg | tail". 

It seems kernel does not like the typo after kernel crash/hung/unclean umount.
Comment 6 Yasuo Ohgaki 2015-08-27 16:57:00 EDT
Kernel 4.1.6
Display output was dead, but I could ssh to the host this time. 

 8月 27 23:07:04 dev.inter.es-i.jp kernel: INFO: trying to register non-static key.
 8月 27 23:07:04 dev.inter.es-i.jp kernel: the code is fine but needs lockdep annotation.
 8月 27 23:07:04 dev.inter.es-i.jp kernel: turning off the locking correctness validator.
 8月 27 23:07:04 dev.inter.es-i.jp kernel: CPU: 5 PID: 6165 Comm: rsync Tainted: G           OE   4.1.6-200.fc22.x86_64+debug #1
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  0000000000000000 000000003ae84776 ffff880168dbb1e8 ffffffff81896965
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  0000000000000000 ffffffff82a7ca70 ffff880168dbb1f8 ffffffff81894f96
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  ffff880168dbb2d8 ffffffff8111269e ffff880168dbb228 ffff8806e0f20000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Call Trace:
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81896965>] dump_stack+0x4c/0x65
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81894f96>] register_lock_class.part.26+0x38/0x3c
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111269e>] __lock_acquire+0x1dbe/0x1e40
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111052d>] ? trace_hardirqs_on+0xd/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125b045>] ? __slab_free+0x1d5/0x2c0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811130f7>] lock_acquire+0xc7/0x2a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737e41>] ? drop_one_stripe+0x41/0xc0 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8189faba>] _raw_spin_lock_irq+0x4a/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737e41>] ? drop_one_stripe+0x41/0xc0 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737e41>] drop_one_stripe+0x41/0xc0 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737f05>] raid5_cache_scan+0x45/0x60 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81209749>] shrink_slab+0x249/0x740
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810eecb5>] ? local_clock+0x25/0x30
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8110cacf>] ? lock_release_holdtime.part.29+0xf/0x200
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8126eb08>] ? mem_cgroup_iter+0x198/0xbf0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d0ac>] shrink_zone+0x2dc/0x2f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d255>] do_try_to_free_pages+0x195/0x430
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d5eb>] try_to_free_pages+0xfb/0x4a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811fe0f1>] __alloc_pages_nodemask+0x801/0xcc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8124ec67>] alloc_pages_current+0x107/0x1c0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259062>] ? new_slab+0x82/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259375>] ? new_slab+0x395/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259062>] new_slab+0x82/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125ab12>] __slab_alloc+0x322/0x680
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d824>] ? kmem_cache_alloc+0x264/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f1d>] ? native_sched_clock+0x2d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] ? btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d824>] kmem_cache_alloc+0x264/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] ? btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03e0cb0>] ? do_async_commit+0x90/0x90 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812a2fed>] alloc_inode+0x1d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812a4d4a>] iget5_locked+0xea/0x240
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03e1070>] ? create_pinned_em+0x130/0x130 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ec93f>] btrfs_iget+0x5f/0x6e0 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81257b6c>] ? rcu_read_lock_sched_held+0x6c/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125b49d>] ? kmem_cache_free+0x36d/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ed83b>] btrfs_lookup_dentry+0x43b/0x550 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ed966>] btrfs_lookup+0x16/0x40 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128ddb6>] ? lookup_slow+0x36/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128cbdd>] lookup_real+0x1d/0x70
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128dd62>] __lookup_hash+0x42/0x60
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128ddc5>] lookup_slow+0x45/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81294a9e>] path_lookupat+0xa1e/0x1090
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d92e>] ? kmem_cache_alloc+0x36e/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812279bf>] ? might_fault+0x5f/0xb0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81295c66>] ? getname_flags+0x56/0x200
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81295137>] filename_lookup+0x27/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81296fc3>] user_path_at_empty+0x63/0xd0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f1d>] ? native_sched_clock+0x2d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f99>] ? sched_clock+0x9/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810eecb5>] ? local_clock+0x25/0x30
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81297041>] user_path_at+0x11/0x20
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128746a>] vfs_fstatat+0x6a/0xd0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111052d>] ? trace_hardirqs_on+0xd/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8118527c>] ? __audit_syscall_entry+0xac/0x100
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81287a81>] SYSC_newlstat+0x31/0x60
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8102cd45>] ? do_audit_syscall_entry+0x55/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8102e09b>] ? syscall_trace_enter_phase1+0x14b/0x1b0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811854c6>] ? __audit_syscall_exit+0x1f6/0x290
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111047d>] ? trace_hardirqs_on_caller+0x13d/0x1e0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff814440e8>] ? trace_hardirqs_on_thunk+0x17/0x19
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81287bbe>] SyS_newlstat+0xe/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff818a076e>] system_call_fastpath+0x12/0x76
 8月 27 23:07:04 dev.inter.es-i.jp kernel: BUG: unable to handle kernel NULL pointer dereference at           (null)
 8月 27 23:07:04 dev.inter.es-i.jp kernel: IP: [<ffffffff814554e1>] __list_del_entry+0x1/0xe0
 8月 27 23:07:04 dev.inter.es-i.jp kernel: PGD 0 
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Oops: 0000 [#1] SMP 
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Modules linked in: bnep bluetooth rfkill vmnet(OE) vmmon(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge ebtable_filter ebtables ip6table_filter ip6_tables ppdev parport_pc parport fuse vmw_vsock_vmci_transport vsock vmw_vmci iTCO_wdt iTCO_vendor_support mxm_wmi raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel i2c_i801 snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_soc_rt5640 snd_soc_rl6231 snd_hda_intel snd_soc_core lpc_ich mfd_core snd_hda_controller snd_compress snd_pcm_dmaengine snd_hda_codec snd_seq mei_me
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  snd_hda_core mei shpchp snd_hwdep snd_seq_device snd_pcm tpm_tis tpm snd_timer snd soundcore regmap_i2c wmi dw_dmac snd_soc_sst_acpi dw_dmac_core i2c_designware_platform acpi_pad i2c_designware_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc ata_generic pata_acpi btrfs 8021q garp stp llc mrp xor raid6_pq crc32c_intel i915 serio_raw i2c_algo_bit drm_kms_helper uas pata_marvell usb_storage e1000e drm ptp pps_core sdhci_acpi sdhci mmc_core video i2c_hid
 8月 27 23:07:04 dev.inter.es-i.jp kernel: CPU: 5 PID: 6165 Comm: rsync Tainted: G           OE   4.1.6-200.fc22.x86_64+debug #1
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 27 23:07:04 dev.inter.es-i.jp kernel: task: ffff8806e0f20000 ti: ffff880168db8000 task.ti: ffff880168db8000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: RIP: 0010:[<ffffffff814554e1>]  [<ffffffff814554e1>] __list_del_entry+0x1/0xe0
 8月 27 23:07:04 dev.inter.es-i.jp kernel: RSP: 0018:ffff880168dbb348  EFLAGS: 00010082
 8月 27 23:07:04 dev.inter.es-i.jp kernel: RAX: fffffffffffffff0 RBX: 0000000000000000 RCX: 0000000000000000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 0000000000000000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: RBP: ffff880168dbb388 R08: 0000000000000000 R09: 0000000000000000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: R10: 0000000000000001 R11: ffff8807f43737d8 R12: ffff8807f4373800
 8月 27 23:07:04 dev.inter.es-i.jp kernel: R13: 00000000ffffffff R14: ffff8807f4373bc8 R15: 0000000000000080
 8月 27 23:07:04 dev.inter.es-i.jp kernel: FS:  00007f42894a0700(0000) GS:ffff8807ff200000(0000) knlGS:0000000000000000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 8月 27 23:07:04 dev.inter.es-i.jp kernel: CR2: 0000000000000000 CR3: 0000000289eb8000 CR4: 00000000001406e0
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Stack:
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  ffff880168dbb388 ffffffffa0737d4f ffff880168dbb388 00000000ffffffff
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  ffff8807f4373800 ffff8807f43737c0 0000000000000980 0000000000000080
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  ffff880168dbb3b8 ffffffffa0737e4b ffff880168dbb3b8 0000000000000053
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Call Trace:
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737d4f>] ? get_free_stripe+0x3f/0xf0 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737e4b>] drop_one_stripe+0x4b/0xc0 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737f05>] raid5_cache_scan+0x45/0x60 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81209749>] shrink_slab+0x249/0x740
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810eecb5>] ? local_clock+0x25/0x30
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8110cacf>] ? lock_release_holdtime.part.29+0xf/0x200
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8126eb08>] ? mem_cgroup_iter+0x198/0xbf0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d0ac>] shrink_zone+0x2dc/0x2f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d255>] do_try_to_free_pages+0x195/0x430
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d5eb>] try_to_free_pages+0xfb/0x4a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811fe0f1>] __alloc_pages_nodemask+0x801/0xcc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8124ec67>] alloc_pages_current+0x107/0x1c0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259062>] ? new_slab+0x82/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259375>] ? new_slab+0x395/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259062>] new_slab+0x82/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125ab12>] __slab_alloc+0x322/0x680
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d824>] ? kmem_cache_alloc+0x264/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f1d>] ? native_sched_clock+0x2d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] ? btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d824>] kmem_cache_alloc+0x264/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] ? btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03e0cb0>] ? do_async_commit+0x90/0x90 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812a2fed>] alloc_inode+0x1d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812a4d4a>] iget5_locked+0xea/0x240
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03e1070>] ? create_pinned_em+0x130/0x130 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ec93f>] btrfs_iget+0x5f/0x6e0 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81257b6c>] ? rcu_read_lock_sched_held+0x6c/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125b49d>] ? kmem_cache_free+0x36d/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ed83b>] btrfs_lookup_dentry+0x43b/0x550 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ed966>] btrfs_lookup+0x16/0x40 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128ddb6>] ? lookup_slow+0x36/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128cbdd>] lookup_real+0x1d/0x70
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128dd62>] __lookup_hash+0x42/0x60
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128ddc5>] lookup_slow+0x45/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81294a9e>] path_lookupat+0xa1e/0x1090
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d92e>] ? kmem_cache_alloc+0x36e/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812279bf>] ? might_fault+0x5f/0xb0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81295c66>] ? getname_flags+0x56/0x200
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81295137>] filename_lookup+0x27/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81296fc3>] user_path_at_empty+0x63/0xd0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f1d>] ? native_sched_clock+0x2d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f99>] ? sched_clock+0x9/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810eecb5>] ? local_clock+0x25/0x30
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81297041>] user_path_at+0x11/0x20
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128746a>] vfs_fstatat+0x6a/0xd0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111052d>] ? trace_hardirqs_on+0xd/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8118527c>] ? __audit_syscall_entry+0xac/0x100
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81287a81>] SYSC_newlstat+0x31/0x60
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8102cd45>] ? do_audit_syscall_entry+0x55/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8102e09b>] ? syscall_trace_enter_phase1+0x14b/0x1b0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811854c6>] ? __audit_syscall_exit+0x1f6/0x290
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111047d>] ? trace_hardirqs_on_caller+0x13d/0x1e0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff814440e8>] ? trace_hardirqs_on_thunk+0x17/0x19
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81287bbe>] SyS_newlstat+0xe/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff818a076e>] system_call_fastpath+0x12/0x76
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Code: 00 00 4d 89 e9 4d 89 e0 48 89 d9 48 c7 c2 b0 1c ca 81 be 24 00 00 00 48 c7 c7 9a 09 ca 81 31 c0 e8 55 c0 c5 ff e9 62 ff ff ff 55 <48> 8b 07 48 b9 00 01 10 00 00 00 ad de 48 8b 57 08 48 89 e5 48 
 8月 27 23:07:04 dev.inter.es-i.jp kernel: RIP  [<ffffffff814554e1>] __list_del_entry+0x1/0xe0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  RSP <ffff880168dbb348>
 8月 27 23:07:04 dev.inter.es-i.jp kernel: CR2: 0000000000000000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: ---[ end trace 6d9197bcdcce5470 ]---
 8月 27 23:07:04 dev.inter.es-i.jp kernel: BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:21
 8月 27 23:07:04 dev.inter.es-i.jp kernel: in_atomic(): 1, irqs_disabled(): 1, pid: 6165, name: rsync
 8月 27 23:07:04 dev.inter.es-i.jp kernel: INFO: lockdep is turned off.
 8月 27 23:07:04 dev.inter.es-i.jp kernel: irq event stamp: 144457766
 8月 27 23:07:04 dev.inter.es-i.jp kernel: hardirqs last  enabled at (144457765): [<ffffffff8189fd16>] _raw_spin_unlock_irqrestore+0x36/0x70
 8月 27 23:07:04 dev.inter.es-i.jp kernel: hardirqs last disabled at (144457766): [<ffffffff8189fa8f>] _raw_spin_lock_irq+0x1f/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel: softirqs last  enabled at (144455568): [<ffffffff810b7c03>] __do_softirq+0x3b3/0x670
 8月 27 23:07:04 dev.inter.es-i.jp kernel: softirqs last disabled at (144455563): [<ffffffff810b8145>] irq_exit+0x145/0x150
 8月 27 23:07:04 dev.inter.es-i.jp kernel: CPU: 5 PID: 6165 Comm: rsync Tainted: G      D    OE   4.1.6-200.fc22.x86_64+debug #1
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z97 Extreme4, BIOS P2.10 05/12/2015
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  0000000000000000 000000003ae84776 ffff880168dbaf98 ffffffff81896965
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  0000000000000000 ffff8806e0f20000 ffff880168dbafc8 ffffffff810e1929
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  ffffffff811277c5 ffffffff81c680c3 0000000000000015 0000000000000000
 8月 27 23:07:04 dev.inter.es-i.jp kernel: Call Trace:
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81896965>] dump_stack+0x4c/0x65
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810e1929>] ___might_sleep+0x189/0x250
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811277c5>] ? vprintk_emit+0x375/0x650
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810e1a3d>] __might_sleep+0x4d/0x90
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8189d8da>] down_read+0x2a/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810c56d3>] exit_signals+0x33/0x160
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810b4a89>] do_exit+0xc9/0xc60
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811284cf>] ? kmsg_dump+0x15f/0x1c0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811283a4>] ? kmsg_dump+0x34/0x1c0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81020ccf>] oops_end+0x9f/0xe0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8107492f>] no_context+0x13f/0x3b0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81074c30>] __bad_area_nosemaphore+0x90/0x220
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81074dd3>] bad_area_nosemaphore+0x13/0x20
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81075116>] __do_page_fault+0xf6/0x4c0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111269e>] ? __lock_acquire+0x1dbe/0x1e40
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8107550f>] do_page_fault+0x2f/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff818a13c7>] ? native_iret+0x7/0x7
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff818a2828>] page_fault+0x28/0x30
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff814554e1>] ? __list_del_entry+0x1/0xe0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737d4f>] ? get_free_stripe+0x3f/0xf0 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737e4b>] drop_one_stripe+0x4b/0xc0 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa0737f05>] raid5_cache_scan+0x45/0x60 [raid456]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81209749>] shrink_slab+0x249/0x740
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810eecb5>] ? local_clock+0x25/0x30
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8110cacf>] ? lock_release_holdtime.part.29+0xf/0x200
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8126eb08>] ? mem_cgroup_iter+0x198/0xbf0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d0ac>] shrink_zone+0x2dc/0x2f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d255>] do_try_to_free_pages+0x195/0x430
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8120d5eb>] try_to_free_pages+0xfb/0x4a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811fe0f1>] __alloc_pages_nodemask+0x801/0xcc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8124ec67>] alloc_pages_current+0x107/0x1c0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259062>] ? new_slab+0x82/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259375>] ? new_slab+0x395/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81259062>] new_slab+0x82/0x4f0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125ab12>] __slab_alloc+0x322/0x680
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d824>] ? kmem_cache_alloc+0x264/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f1d>] ? native_sched_clock+0x2d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] ? btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d824>] kmem_cache_alloc+0x264/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] ? btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03e0cb0>] ? do_async_commit+0x90/0x90 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03f1173>] btrfs_alloc_inode+0x23/0x220 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812a2fed>] alloc_inode+0x1d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812a4d4a>] iget5_locked+0xea/0x240
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03e1070>] ? create_pinned_em+0x130/0x130 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ec93f>] btrfs_iget+0x5f/0x6e0 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81257b6c>] ? rcu_read_lock_sched_held+0x6c/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125b49d>] ? kmem_cache_free+0x36d/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ed83b>] btrfs_lookup_dentry+0x43b/0x550 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffffa03ed966>] btrfs_lookup+0x16/0x40 [btrfs]
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128ddb6>] ? lookup_slow+0x36/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128cbdd>] lookup_real+0x1d/0x70
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128dd62>] __lookup_hash+0x42/0x60
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128ddc5>] lookup_slow+0x45/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81294a9e>] path_lookupat+0xa1e/0x1090
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8125d92e>] ? kmem_cache_alloc+0x36e/0x3a0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff812279bf>] ? might_fault+0x5f/0xb0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81295c66>] ? getname_flags+0x56/0x200
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81295137>] filename_lookup+0x27/0xc0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81296fc3>] user_path_at_empty+0x63/0xd0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f1d>] ? native_sched_clock+0x2d/0xa0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81027f99>] ? sched_clock+0x9/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff810eecb5>] ? local_clock+0x25/0x30
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81297041>] user_path_at+0x11/0x20
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8128746a>] vfs_fstatat+0x6a/0xd0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111052d>] ? trace_hardirqs_on+0xd/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8118527c>] ? __audit_syscall_entry+0xac/0x100
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81287a81>] SYSC_newlstat+0x31/0x60
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8102cd45>] ? do_audit_syscall_entry+0x55/0x80
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8102e09b>] ? syscall_trace_enter_phase1+0x14b/0x1b0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff811854c6>] ? __audit_syscall_exit+0x1f6/0x290
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff8111047d>] ? trace_hardirqs_on_caller+0x13d/0x1e0
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff814440e8>] ? trace_hardirqs_on_thunk+0x17/0x19
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff81287bbe>] SyS_newlstat+0xe/0x10
 8月 27 23:07:04 dev.inter.es-i.jp kernel:  [<ffffffff818a076e>] system_call_fastpath+0x12/0x76
 8月 27 23:07:04 dev.inter.es-i.jp kernel: note: rsync[6165] exited with preempt_count 1
 8月 27 23:07:04 dev.inter.es-i.jp /usr/share/back[6061]: backintime (root): WARNING: Command "rsync -rtDH --links --no-p --no-g --no-o  --delete --delete-excluded  -i --dry-run --out-format="BACKINTIME: %i %n%L" --chmod=Du+wx  --exclude="/backup" --exclude="/root/.local/share/backintime" --exclude="/tmp/backintime" --include="/home/yohgaki/" --include="/home/" --exclude=".gvfs" --exclude=".cache*" --exclude="[Cc]ache*" --exclude=".thumbnails*" --exclude="[Tt]rash*" --exclude="*.backup*" --exclude="*~" --exclude=".dropbox*" --exclude="/proc/*" --exclude="/sys/*" --exclude="/dev/*" --exclude="/run/*" --exclude="/home/yohgaki/vmware*" --exclude=".ccache" --include="/home/yohgaki/**" --exclude="*" / "/backup/backintime/dev.inter.es-i.jp/root/1/20150827-052523-165/backup/"" returns 3072
Comment 7 Laura Abbott 2015-08-27 18:48:54 EDT
So https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=49895bcc7e566ba455eb2996607d6fbd3447ce16 looks like the same report you have (drop_one_stripe called from raid5_cache_scan) and it's queued for stable but it is related to https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=2d5b569b665 which has a note to be delayed for 2 weeks after the release of 4.2 . 

I'm not really eager to bring patches with caveats like that into the Fedora stable tree early without an okay from the author. 4.2 is going to come out really soon so by the time those patches actually ended up in a 4.1.x tree F22 would probably be on 4.2 anyway. After F22 gets 4.2 we can re-evaluate if this bug is still an issue.
Comment 8 Stephen Champion 2015-09-08 09:16:28 EDT
This appears to be quite severe.

I have a home system with an xfs FS on three disk raid 5 served via smb to hosts doing backups.  It has been crashing daily for more than a week, shortly after backups start.  This started around the time it was upgraded to 4.1.6-200.fc22.x86_64.

I've finally gotten kdump working well enough to get the dmesg.  I should be able to provide more detail if it is helpful, just have to get it fully functional.

[85745.025952] general protection fault: 0000 [#1] SMP 
[85745.027222] Modules linked in: fuse xt_multiport nf_nat_ftp nf_conntrack_ftp nf_nat_irc nf_conntrack_irc nf_log_ipv4 nf_log_common xt_LOG ip6t_REJECT xt_hashlimit nf_reject_ipv6 xt_recent nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_nat_ipv4 nf_nat ip6table_filter nf_conntrack ip6_tables it87 hwmon_vid xfs snd_hda_codec_hdmi libcrc32c kvm_amd kvm eeepc_wmi asus_wmi sparse_keymap rfkill video edac_core edac_mce_amd k10temp vfat fat snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer drm snd soundcore sp5100_tco shpchp i2c_piix4 acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd binfmt_misc grace sunrpc raid456 async_raid6_recov async_memcpy async_pq async_xor
[85745.037399]  xor async_tx raid6_pq raid1 mxm_wmi firewire_ohci serio_raw r8169 firewire_core mii crc_itu_t tg3 ptp pps_core wmi
[85745.039895] CPU: 0 PID: 25125 Comm: kworker/u16:1 Tainted: P           OE   4.1.6-200.fc22.x86_64 #1
[85745.042360] Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX, BIOS 1604 10/16/2012
[85745.044817] Workqueue: writeback bdi_writeback_workfn (flush-9:20)
[85745.046040] task: ffff880429e44520 ti: ffff880150830000 task.ti: ffff880150830000
[85745.047258] RIP: 0010:[<ffffffffa00e4630>]  [<ffffffffa00e4630>] __find_stripe+0x30/0xc0 [raid456]
[85745.049670] RSP: 0018:ffff880150833578  EFLAGS: 00010006
[85745.050861] RAX: 000088003e923000 RBX: 0000000000000000 RCX: ffff880150833610
[85745.052046] RDX: 0000000000000400 RSI: 00000000bb563400 RDI: ffff88042946e800
[85745.053211] RBP: ffff880150833598 R08: 0000000000000000 R09: 0000000000000001
[85745.054326] R10: 0000000000000400 R11: 0000000000000001 R12: 00000000bb563400
[85745.055454] R13: ffff88042946e800 R14: 00000000bb563400 R15: 0000000000000000
[85745.056563] FS:  00007f5001f16700(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000
[85745.057666] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[85745.058750] CR2: 00007f239007c000 CR3: 0000000423c76000 CR4: 00000000000006f0
[85745.059830] Stack:
[85745.060878]  0000000000000000 ffff88042946e800 ffff88042946e808 ffff88042946ea18
[85745.061942]  ffff880150833658 ffffffffa00e81a7 ffff88042962c298 0000000000000246
[85745.062993]  ffff880150833608 ffffffff810e4ca7 ffff880400000000 0000000000000287
[85745.063985] Call Trace:
[85745.064983]  [<ffffffffa00e81a7>] get_active_stripe+0x137/0x710 [raid456]
[85745.065984]  [<ffffffff810e4ca7>] ? prepare_to_wait_event+0x87/0x100
[85745.066965]  [<ffffffff810e4884>] ? prepare_to_wait+0x54/0x80
[85745.067919]  [<ffffffff810e4d90>] ? wake_atomic_t_function+0x70/0x70
[85745.068840]  [<ffffffffa00ecca9>] make_request+0x1a9/0xca0 [raid456]
[85745.069781]  [<ffffffff811af91d>] ? mempool_alloc_slab+0x1d/0x30
[85745.070705]  [<ffffffff810e4d90>] ? wake_atomic_t_function+0x70/0x70
[85745.071614]  [<ffffffff815fdb0c>] md_make_request+0xec/0x230
[85745.072470]  [<ffffffff8137d3b0>] generic_make_request+0xe0/0x130
[85745.073340]  [<ffffffff8137d478>] submit_bio+0x78/0x180
[85745.074210]  [<ffffffffa041cb61>] xfs_submit_ioend_bio.isra.14+0x31/0x40 [xfs]
[85745.075032]  [<ffffffffa041cc5e>] xfs_submit_ioend+0xee/0x130 [xfs]
[85745.075870]  [<ffffffffa041d0ac>] xfs_vm_writepage+0x2cc/0x680 [xfs]
[85745.076674]  [<ffffffff811b7997>] __writepage+0x17/0x40
[85745.077421]  [<ffffffff811b817d>] write_cache_pages+0x22d/0x500
[85745.078187]  [<ffffffff811b7980>] ? bdi_set_max_ratio+0x90/0x90
[85745.078937]  [<ffffffff811b849d>] generic_writepages+0x4d/0x80
[85745.079646]  [<ffffffffa041b8c2>] xfs_vm_writepages+0x42/0x50 [xfs]
[85745.080366]  [<ffffffff811ba24e>] do_writepages+0x1e/0x30
[85745.081039]  [<ffffffff81256be5>] __writeback_single_inode+0x45/0x2c0
[85745.081740]  [<ffffffff8125722b>] writeback_sb_inodes+0x1db/0x4c0
[85745.082392]  [<ffffffff812575b2>] __writeback_inodes_wb+0xa2/0xd0
[85745.083057]  [<ffffffff81257833>] wb_writeback+0x253/0x2e0
[85745.083681]  [<ffffffff81246a14>] ? get_nr_inodes+0x54/0x80
[85745.084330]  [<ffffffff8125a33e>] bdi_writeback_workfn+0x2de/0x470
[85745.084951]  [<ffffffff810136d6>] ? __switch_to+0x216/0x5d0
[85745.085598]  [<ffffffff810baadb>] process_one_work+0x1bb/0x410
[85745.086214]  [<ffffffff810bad83>] worker_thread+0x53/0x480
[85745.086860]  [<ffffffff810bad30>] ? process_one_work+0x410/0x410
[85745.087462]  [<ffffffff810bad30>] ? process_one_work+0x410/0x410
[85745.088076]  [<ffffffff810c0bf8>] kthread+0xd8/0xf0
[85745.088647]  [<ffffffff810c0b20>] ? kthread_worker_fn+0x180/0x180
[85745.089246]  [<ffffffff817a2322>] ret_from_fork+0x42/0x70
[85745.089797]  [<ffffffff810c0b20>] ? kthread_worker_fn+0x180/0x180
[85745.090378] Code: 55 48 89 e5 41 55 41 54 53 49 89 fd 49 89 f4 89 d3 48 83 ec 08 f6 05 99 16 01 00 04 75 5e 49 8b 45 00 4c 89 e2 81 e2 f8 0f 00 00 <48> 8b 04 02 48 85 c0 75 2d 31 db f6 05 50 16 01 00 04 75 57 48 
[85745.092212] RIP  [<ffffffffa00e4630>] __find_stripe+0x30/0xc0 [raid456]
[85745.092850]  RSP <ffff880150833578>
Comment 9 Stephen Champion 2015-09-08 21:19:14 EDT
> This started around the time it was upgraded to 4.1.6-200.fc22.x86_64.

The system was upgraded from kernel-4.0.8-300.fc22.x86_64.

To be clear to anyone who comes across this, it looks like the bug was introduced by

  edbe83ab4c27 md/raid5: allow the stripe_cache to grow and shrink.

in which case all 4.1 tags are affected.
v4.2-rc6 and later contain both of the relevant fixes Laura identified.

The two week delay is a recommendation for accepting the patch to 4.1 maintenance branches.

When a 4.2 kernel for FC22 is available, I would be delighted to test it.
Comment 10 Stephen Champion 2015-09-10 15:39:05 EDT
Using 4.2.0-300.fc23, I have confirmed that this problem is resolved.
Comment 11 Justin M. Forbes 2015-10-20 15:34:12 EDT
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 22 kernel bugs.

Fedora 22 has now been rebased to 4.2.3-200.fc22.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 23, and are still experiencing this issue, please change the version to Fedora 23.

If you experience different issues, please open a new bug report for those.
Comment 12 Stephen Champion 2015-10-29 00:35:45 EDT
The system that was tripping over this daily on 4.1 kernels has been running for more than a week with 4.2.3-200.fc22.x86_64.

This BZ can be closed.
Comment 13 Josh Boyer 2015-10-29 07:32:39 EDT
Thanks for letting us know.

Note You need to log in before you can comment on or make changes to this bug.