Hide Forgot
I've received the following oops a couple of times, always when my backups kick off just past midnight. This leads to a bunch of follow-on oopses. The machine is still up but disk access just hangs so eventually the machine becomes useless. When this happens, the machine is practically idle except for the backup run. There should be no memory pressure; the machine has 64GB of RAM and isn't running anything other than one ssh login and a few shells. Note that there are a few lines before the actual oops for context some of which I'm pretty sure are important. I have more complete logs and can provide them if you would like to see them. Nov 22 00:05:01 epithumia.math.uh.edu systemd[1]: Started Amanda Backup System (129.7.128.43:571). Nov 22 00:05:03 epithumia.math.uh.edu systemd[1]: Started Process Core Dump (PID 9986/UID 0). Nov 22 00:05:03 epithumia.math.uh.edu systemd-coredump[9987]: Failed to compress /var/lib/systemd/coredump/.#core.xfsdump.0.160232b14aef47deb209233236226794.9985.1479794703000000000000.lz44b0306156376cc56: Invalid argument Nov 22 00:05:03 epithumia.math.uh.edu systemd-coredump[9987]: Failed to generate stack trace: invalid `Elf' handle Nov 22 00:05:03 epithumia.math.uh.edu systemd-coredump[9987]: Process 9985 (xfsdump) of user 0 dumped core. Nov 22 00:06:01 epithumia.math.uh.edu systemd[1]: Started Usertrack client service (129.7.128.201:55440). Nov 22 00:06:01 epithumia.math.uh.edu systemd[1]: Started Amanda Backup System (129.7.128.43:571). Nov 22 00:06:02 epithumia.math.uh.edu systemd[1]: Started Process Core Dump (PID 10084/UID 0). Nov 22 00:06:02 epithumia.math.uh.edu systemd-coredump[10085]: Failed to compress /var/lib/systemd/coredump/.#core.xfsrestore.33.160232b14aef47deb209233236226794.10079.1479794762000000000000.lz4c8b50cdd84ea01ca: Invalid argument Nov 22 00:06:02 epithumia.math.uh.edu systemd-coredump[10085]: Failed to generate stack trace: invalid `Elf' handle Nov 22 00:06:02 epithumia.math.uh.edu systemd-coredump[10085]: Process 10079 (xfsrestore) of user 33 dumped core. Nov 22 00:06:10 epithumia.math.uh.edu systemd[1]: Started Amanda Backup System (129.7.128.43:585). Nov 22 00:06:18 epithumia.math.uh.edu systemd[1]: Started Amanda Backup System (129.7.128.43:585). Nov 22 00:08:01 epithumia.math.uh.edu systemd[1]: Started Usertrack client service (129.7.128.201:55786). Nov 22 00:10:01 epithumia.math.uh.edu systemd[1]: Started Usertrack client service (129.7.128.201:56132). Nov 22 00:10:36 epithumia.math.uh.edu systemd-logind[1126]: New session 142 of user tibbs. Nov 22 00:10:36 epithumia.math.uh.edu systemd[1]: Started Session 142 of user tibbs. Nov 22 00:12:01 epithumia.math.uh.edu systemd[1]: Started Usertrack client service (129.7.128.201:56480). Nov 22 00:12:12 epithumia.math.uh.edu systemd[1]: Started Amanda Backup System (129.7.128.43:585). Nov 22 00:12:13 epithumia.math.uh.edu kernel: page:ffffea0000326180 count:0 mapcount:0 mapping: (null) index:0x0 Nov 22 00:12:13 epithumia.math.uh.edu kernel: flags: 0x1ffffc00000000() Nov 22 00:12:13 epithumia.math.uh.edu kernel: page dumped because: VM_BUG_ON_PAGE(page_ref_count(page) == 0) Nov 22 00:12:13 epithumia.math.uh.edu kernel: ------------[ cut here ]------------ Nov 22 00:12:13 epithumia.math.uh.edu kernel: kernel BUG at ./include/linux/mm.h:445! Nov 22 00:12:13 epithumia.math.uh.edu kernel: invalid opcode: 0000 [#1] SMP Nov 22 00:12:13 epithumia.math.uh.edu kernel: Modules linked in: snd_usb_audio snd_usbmidi_lib snd_rawmidi nfsv3 nfsv4 dns_resolver nfs fscache rfcomm cts rpcsec_gss_krb5 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack cmac bnep nct6775 hwmon_vid b43 mac80211 intel_rapl snd_hda_codec_realtek sb_edac snd_hda_codec_generic edac_core x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp vfat fat kvm_intel cfg80211 kvm ssb mmc_core iTCO_wdt iTCO_vendor_support mxm_wmi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel btusb btrtl intel_cstate btbcm btintel intel_uncore bluetooth intel_rapl_perf rfkill snd_hda_intel snd_hda_codec joydev i2c_i801 snd_hda_core i2c_smbus snd_hwdep snd_seq snd_seq_device igb snd_pcm e1000e snd_timer bcma snd dca mei_me soundcore ptp mei lpc_ich pps_core shpchp fjes wmi tpm_tis Nov 22 00:12:13 epithumia.math.uh.edu kernel: tpm_tis_core tpm nfsd nfs_acl lockd auth_rpcgss grace binfmt_misc sunrpc xfs libcrc32c hid_logitech_hidpp hid_logitech_dj amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper ttm crc32c_intel drm serio_raw nvme nvme_core uas usb_storage Nov 22 00:12:13 epithumia.math.uh.edu kernel: CPU: 1 PID: 12234 Comm: xfsdump Not tainted 4.8.6-201.fc24.x86_64 #1 Nov 22 00:12:13 epithumia.math.uh.edu kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99E-ITX/ac, BIOS P3.40 08/03/2016 Nov 22 00:12:13 epithumia.math.uh.edu kernel: task: ffff97c75bf13d00 task.stack: ffff97cde5c00000 Nov 22 00:12:13 epithumia.math.uh.edu kernel: RIP: 0010:[<ffffffff951c1a94>] [<ffffffff951c1a94>] __free_pages+0x34/0x40 Nov 22 00:12:13 epithumia.math.uh.edu kernel: RSP: 0018:ffff97cde5c03a38 EFLAGS: 00010246 Nov 22 00:12:13 epithumia.math.uh.edu kernel: RAX: 000000000000003e RBX: ffff97c76e25f200 RCX: 0000000000000006 Nov 22 00:12:13 epithumia.math.uh.edu kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff97d39f24e040 Nov 22 00:12:13 epithumia.math.uh.edu kernel: RBP: ffff97cde5c03a38 R08: 0000000000035a75 R09: 0000000000000005 Nov 22 00:12:13 epithumia.math.uh.edu kernel: R10: 0000000000000000 R11: 0000000000000544 R12: 0000000000000001 Nov 22 00:12:13 epithumia.math.uh.edu kernel: R13: ffffffffc060dde6 R14: ffff97cde5c03b28 R15: 0000000000000001 Nov 22 00:12:13 epithumia.math.uh.edu kernel: FS: 00007f08fcb65780(0000) GS:ffff97d39f240000(0000) knlGS:0000000000000000 Nov 22 00:12:13 epithumia.math.uh.edu kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 22 00:12:13 epithumia.math.uh.edu kernel: CR2: 00007f08fb80a000 CR3: 0000000adb50b000 CR4: 00000000003406e0 Nov 22 00:12:13 epithumia.math.uh.edu kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Nov 22 00:12:13 epithumia.math.uh.edu kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Nov 22 00:12:13 epithumia.math.uh.edu kernel: Stack: Nov 22 00:12:13 epithumia.math.uh.edu kernel: ffff97cde5c03a60 ffffffffc060cef3 ffff97d393a40900 0000000000000000 Nov 22 00:12:13 epithumia.math.uh.edu kernel: 0000000000010015 ffff97cde5c03aa0 ffffffffc060dde6 ffff97c76e25f200 Nov 22 00:12:13 epithumia.math.uh.edu kernel: 0000000000010015 0000000000000001 0000000000010014 ffff97d393a40900 Nov 22 00:12:13 epithumia.math.uh.edu kernel: Call Trace: Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc060cef3>] xfs_buf_free+0x73/0x130 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc060dde6>] xfs_buf_get_map+0x206/0x250 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc060e7dd>] xfs_buf_read_map+0x2d/0x170 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc060e972>] xfs_buf_readahead_map+0x52/0x70 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc05e1c83>] xfs_btree_reada_bufs+0x63/0x80 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc06239fa>] xfs_bulkstat_ichunk_ra.isra.5+0xea/0x140 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc05defa2>] ? xfs_btree_rec_addr+0x12/0x20 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc0624069>] xfs_bulkstat+0x2e9/0x660 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc0623d50>] ? xfs_bulkstat_one_int+0x300/0x300 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc0618754>] xfs_ioc_bulkstat+0xe4/0x1a0 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffffc061b21d>] xfs_file_ioctl+0x9fd/0xaf0 [xfs] Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffff951cc35a>] ? lru_cache_add+0x3a/0x80 Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffff951cc53c>] ? lru_cache_add_active_or_unevictable+0x4c/0xf0 Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffff951f556f>] ? handle_mm_fault+0xfaf/0x1420 Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffff95264dd3>] do_vfs_ioctl+0xa3/0x5f0 Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffff95265399>] SyS_ioctl+0x79/0x90 Nov 22 00:12:13 epithumia.math.uh.edu kernel: [<ffffffff95802572>] entry_SYSCALL_64_fastpath+0x1a/0xa4 Nov 22 00:12:13 epithumia.math.uh.edu kernel: Code: 48 89 e5 85 c0 74 18 f0 ff 4f 1c 75 09 85 f6 74 07 e8 a1 e1 ff ff 5d c3 e8 1a fd ff ff 5d c3 48 c7 c6 f0 dd c1 95 e8 9c c0 02 00 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 85 ff 75 Nov 22 00:12:13 epithumia.math.uh.edu kernel: RIP [<ffffffff951c1a94>] __free_pages+0x34/0x40 Nov 22 00:12:13 epithumia.math.uh.edu kernel: RSP <ffff97cde5c03a38>
Ok Jason said he had 2 instances of this. Just for reference: First page dump: page:ffffea003f6aed80 count:0 mapcount:0 mapping: (null) index:0x1 flags: 0x2ffffc00000000() Second page dump: page:ffffea0000326180 count:0 mapcount:0 mapping: (null) index:0x0 flags: 0x1ffffc00000000()
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 24 kernel bugs. Fedora 25 has now been rebased to 4.10.9-100.fc24. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 26, and are still experiencing this issue, please change the version to Fedora 26. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 2 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.