Created attachment 948353 [details]
File: dmesg

Description of problem:
With Fedora 21 (3.17 kernel) I spotted several filesystem problems (F20 with 3.16 was OK), such as:

Nov 16 08:22:36 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 369, block bitmap and bg descriptor inconsistent: 23619 vs 23620 free clusters
Nov 16 08:22:36 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 328, block bitmap and bg descriptor inconsistent: 31556 vs 31172 free clusters
Nov 16 08:22:40 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 326, block bitmap and bg descriptor inconsistent: 32130 vs 32129 free clusters
Nov 16 08:23:24 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 322, block bitmap and bg descriptor inconsistent: 30067 vs 30033 free clusters
Nov 16 08:23:24 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 329, block bitmap and bg descriptor inconsistent: 30858 vs 31364 free clusters
Nov 16 08:24:54 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 325, block bitmap and bg descriptor inconsistent: 27884 vs 27883 free clusters
Nov 16 08:25:42 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 79, block bitmap and bg descriptor inconsistent: 9719 vs 9698 free clusters
Nov 16 08:40:19 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 368, block bitmap and bg descriptor inconsistent: 21992 vs 21993 free clusters
Nov 16 08:40:20 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 323, block bitmap and bg descriptor inconsistent: 32133 vs 32135 free clusters
Nov 16 09:01:25 assam kernel: EXT4-fs error (device dm-1): ext4_lookup:1448: inode #2622888: comm chronyd: deleted inode referenced: 2624199
Nov 16 09:01:25 assam kernel: EXT4-fs error (device dm-1): ext4_lookup:1448: inode #2622888: comm chronyd: deleted inode referenced: 2624199
Nov 16 09:13:35 assam kernel: EXT4-fs error (device dm-1): ext4_mb_free_metadata:4598: group 321, block 10521946:Block already on to-be-freed list
Nov 16 09:27:06 assam kernel: EXT4-fs error (device dm-1): ext4_lookup:1448: inode #2622796: comm alsactl: deleted inode referenced: 2624355

Before the backtace I got this:

Nov 16 12:54:55 assam kernel: EXT4-fs error (device dm-1): mb_free_blocks:1450: group 327, block 10717753:freeing already freed block (bit 2617); block bitmap corrupt.
Nov 16 12:54:55 assam kernel: EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 327, block bitmap and bg descriptor inconsistent: 32484 vs 32486 free clusters

And after 10 minutes this:

Nov 16 13:04:53 assam kernel: EXT4-fs error (device dm-1): ext4_lookup:1448: inode #2622888: comm chronyd: deleted inode referenced: 2624199
Nov 16 13:04:53 assam kernel: EXT4-fs error (device dm-1): ext4_lookup:1448: inode #2622888: comm chronyd: deleted inode referenced: 2624199

Some of the errors seem to be related to `chronyd'.

This week I had to fix couple of the EXT4 errors by `fsck', while I lost some files from root, but the corruptions are still comming. SMART says "disk is OK." I'll probably need to downgrade to 3.16.

If there's anything I can help with, just let me know.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.3.0
cmdline:        BOOT_IMAGE=/vmlinuz-3.17.2-300.fc21.x86_64 root=/dev/mapper/vg_assam-lv_root ro rd.lvm.lv=vg_assam/lv_root rd.md=0 rd.lvm.lv=vg_assam/lv_swap rd.luks.uuid=luks-077ce69c-df71-423e-9f7f-3a26134793e0 SYSFONT=True KEYTABLE=us LANG=en_US.UTF-8 rd.dm=0 rhgb quiet resume=UUID=7525d3d1-e34a-4c52-89d3-dcb2fe51f4c1 radeon.dpm=1
kernel:         3.17.2-300.fc21.x86_64
runlevel:       N 5
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 6 PID: 906 at fs/inode.c:282 drop_nlink+0x41/0x50()
Modules linked in: ccm alx uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media rfcomm fuse xt_CHECKSUM ipt_MASQUERADE tun nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw bnep rtsx_usb_ms memstick snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel ath3k btusb arc4 ath9k bluetooth snd_hda_controller ath9k_common ath9k_hw snd_hda_codec ath snd_hwdep mac80211 x86_pkg_temp_thermal
 snd_seq kvm_intel snd_seq_device snd_pcm kvm cfg80211 toshiba_acpi sparse_keymap snd_timer rfkill wmi iTCO_wdt iTCO_vendor_support mei_me video toshiba_bluetooth i2c_i801 snd mei lpc_ich serio_raw shpchp soundcore coretemp i2c_dev binfmt_misc dm_crypt radeon rtsx_usb_sdmmc mmc_core i2c_algo_bit drm_kms_helper ttm crct10dif_pclmul drm crc32_pclmul crc32c_intel ghash_clmulni_intel rtsx_usb mfd_core mdio [last unloaded: alx]
CPU: 6 PID: 906 Comm: alsactl Not tainted 3.17.2-300.fc21.x86_64 #1
Hardware name: TOSHIBA SATELLITE L855/Portable PC, BIOS 6.60 01/14/2013
 0000000000000000 00000000c60fb863 ffff8801c479bbb8 ffffffff8173db01
 0000000000000000 ffff8801c479bbf0 ffffffff81096e8d ffff8800a6b6a820
 ffff8801c479bc40 ffff8801c4caf030 ffff8801037eb048 0000000000000000
Call Trace:
 [<ffffffff8173db01>] dump_stack+0x45/0x56
 [<ffffffff81096e8d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff81096fba>] warn_slowpath_null+0x1a/0x20
 [<ffffffff81227ba1>] drop_nlink+0x41/0x50
 [<ffffffff812a042b>] ext4_dec_count.isra.21+0x1b/0x30
 [<ffffffff812a575a>] ext4_rename+0x39a/0x6d0
 [<ffffffff812a5aad>] ext4_rename2+0x1d/0x40
 [<ffffffff8121b023>] vfs_rename+0x4c3/0x780
 [<ffffffff8121b7c8>] SYSC_renameat2+0x4e8/0x560
 [<ffffffff811c7049>] ? vma_rb_erase+0x129/0x250
 [<ffffffff8121eb6e>] SyS_rename+0x1e/0x20
 [<ffffffff81744c69>] system_call_fastpath+0x16/0x1b

Created attachment 957993 [details]
EXT4-fs errors

lonelywoolf: Can you also see filesystem corruptions as I do?

Looking at 3.17.3 there may be a fix for this:

commit 5be56bf0f1040b9b05cc3fc1ed0e9e4816385c74
Author: Theodore Ts'o <tytso>
Date:   Sun Oct 5 22:47:07 2014 -0400

    ext4: don't orphan or truncate the boot loader inode
    
    commit e2bfb088fac03c0f621886a04cffc7faa2b49b1d upstream.
    
    The boot loader inode (inode #5) should never be visible in the
    directory hierarchy, but it's possible if the file system is corrupted
    that there will be a directory entry that points at inode #5.  In
    order to avoid accidentally trashing it, when such a directory inode
    is opened, the inode will be marked as a bad inode, so that it's not
    possible to modify (or read) the inode from userspace.
    
    Unfortunately, when we unlink this (invalid/illegal) directory entry,
    we will put the bad inode on the ophan list, and then when try to
    unlink the directory, we don't actually remove the bad inode from the
    orphan list before freeing in-memory inode structure.  This means the
    in-memory orphan list is corrupted, leading to a kernel oops.
    
    In addition, avoid truncating a bad inode in ext4_destroy_inode(),
    since truncating the boot loader inode is not a smart thing to do.



I'll give it a try.

(In reply to Michal Nowak from comment #4)
> lonelywoolf: Can you also see filesystem corruptions as I do?

Yes, root filesystem is corrupting. Especially after sleep mode. Just backing up my data every hour now.

After an hour with 3.17.3 kernel from Koji I am good, no corruptions. Perhaps you could upgrade to it as well, so you can verify it's fixed?

Will update now. Need few days for testing.

I can't reproduce this bug with 3.17.3 kernel. All works as expected. Tested suspend/hibernate, tested filesystem under heavy load suc as copying/writing, burned in with virtual machines. This kernel resolved some other bugs in other software for me, such as unexpected behavior of software under heavy CPU load. Gouing to leave good karma for this kernel. Thanks for the advice about updating to this kernel.

Created attachment 958900 [details]
EXT4-fs errors with 3.17.3 kernel

I was wrong in assuming the bug is gone, the problem is still present even with 3.17.3:

EXT4-fs error (device dm-1): ext4_mb_generate_buddy:757: group 321, block bitmap and bg descriptor inconsistent: 24882 vs 24881 free clusters

See the attached log.

lonewolf: Can you see it as well?

No,I have absolutelly clean filesystems. Just tested now with reboot to single-user. 2 days uptime after update without any issues. Should I reopen this bug for you?

Just for the further investigation: Are you use SSD? Which options are you use in /etc/fstab? You use LVM/dm-crypt? If so, please attach config files.

And some note: before any tests do this:
1. Boot in single user mode.
2. Modify mount options in /etc/fstab to read-only for root fs.
3. Reboot in single user mode.
4. check all FS in your system.
5. reboot in single-user
6. remount root fs with read-write
7. modify /etc/fstab to read-write root fs.
8. reboot and enjoy.

After this steps I have no any similar problem. Also you may check your RAM with memtest.

(In reply to lonelywoolf from comment #11)
> No,I have absolutelly clean filesystems. Just tested now with reboot to
> single-user. 2 days uptime after update without any issues. Should I reopen
> this bug for you?

I'll file separate bug, thanks for the info.

Description of problem:
Kernel oops happened while reinstalling firefox through yumex (Yum Extender).
This is a *critical error* - it has happened many times and caused a *great deal of data loss*. This is the first time that it has been captured by ABRT.
All previous errors seem to happen after a reboot (when the root filesystem is marked unclean and cannot be repaired without a rescue CD) when the reboot follows an installation via yumex.
Data loss is so great that gigabytes of data on some occasions can be lost and some reinstallation of packages necessary to get the system functioning properly again. It's only the root filesystem that gets corrupted (there's a home filesystem too) - presumably because yumex is installing packages on the root filesystem.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.3.0
cmdline:        BOOT_IMAGE=/vmlinuz-3.18.3-201.fc21.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.md=0 rd.dm=0 rd.luks=0 vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8
kernel:         3.18.3-201.fc21.x86_64
runlevel:       N 5
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 3 PID: 1884 at fs/inode.c:282 drop_nlink+0x49/0x50()
Modules linked in: bnep bluetooth rfkill fuse ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw joydev snd_ctxfi iTCO_wdt gpio_ich coretemp iTCO_vendor_support dcdbas ppdev snd_seq snd_seq_device snd_pcm kvm snd_timer snd soundcore serio_raw i2c_i801 x38_edac edac_core tpm_tis shpchp lpc_ich mfd_core tpm parport_pc parport acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc nouveau video mxm_wmi wmi i2c_algo_bit drm_kms_helper ttm tg3 drm ptp pps_core uas usb_storage
CPU: 3 PID: 1884 Comm: yum_childtask.p Not tainted 3.18.3-201.fc21.x86_64 #1
Hardware name: Dell Inc. Precision WorkStation T3400  /0TP412, BIOS A04 03/21/2008
 0000000000000000 00000000030e3ab1 ffff8800b2f87b48 ffffffff8175da66
 0000000000000000 0000000000000000 ffff8800b2f87b88 ffffffff81099181
 ffff8800b2f87bc8 ffff8800b5110c90 ffff8800b2f87c10 0000000000000000
Call Trace:
 [<ffffffff8175da66>] dump_stack+0x46/0x58
 [<ffffffff81099181>] warn_slowpath_common+0x81/0xa0
 [<ffffffff8109929a>] warn_slowpath_null+0x1a/0x20
 [<ffffffff81231ba9>] drop_nlink+0x49/0x50
 [<ffffffff812ab17b>] ext4_dec_count.isra.21+0x1b/0x30
 [<ffffffff812b0c45>] ext4_rename+0x4e5/0x8b0
 [<ffffffff812b102d>] ext4_rename2+0x1d/0x40
 [<ffffffff81222c52>] vfs_rename+0x3d2/0x860
 [<ffffffff81227537>] SYSC_renameat2+0x587/0x600
 [<ffffffff81217eb8>] ? __sb_start_write+0x58/0x110
 [<ffffffff81021d9c>] ? do_audit_syscall_entry+0x6c/0x70
 [<ffffffff8102363b>] ? syscall_trace_enter_phase1+0x13b/0x1a0
 [<ffffffff812288be>] SyS_rename+0x1e/0x20
 [<ffffffff817647a9>] system_call_fastpath+0x12/0x17

Note that the above (from me) was added to this bug by the ABRT wizard - hopefully it was get noticed even though the bug is closed.

Jonathan: My observation is that 3.16 is good, so you may want to downgrade to it. With 3.17 (but not 3.18) I saw many FS corruptions on the root partition too, but none on the one where /home was; until this issue is fixed perhaps you should move /home away from root partition (just a guess).

Michal
Thanks for the quick response. Downgrading the kernel sounds like a good idea - I've installed kernel-3.16.7-200.fc20, which is the last 3.16 kernel for fc20. I'll see how it goes.
I did an upgrade from Fedora19 (via fedup) to Fedora21 - I never had a problem on 19, which was kernel-3.14 at its highest. The problem started immediately on upgrade to Fedora21, which tallies with kernels 3.17 & 3.18 having this bug.

Description of problem:
Bootup from power off.

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.3.0
cmdline:        BOOT_IMAGE=/vmlinuz-3.17.4-301.fc21.x86_64 root=/dev/mapper/fedora_1-root ro rd.lvm.lv=fedora_1/swap rd.lvm.lv=fedora_1/root rhgb quiet LANG=en_US.UTF-8
kernel:         3.17.4-301.fc21.x86_64
runlevel:       N 5
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 1 PID: 1447 at fs/inode.c:282 drop_nlink+0x41/0x50()
Modules linked in: ip6t_rpfilter ip6t_REJECT xt_conntrack cfg80211 ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm joydev hp_wmi snd_timer coretemp ppdev sparse_keymap gpio_ich rfkill iTCO_wdt iTCO_vendor_support kvm serio_raw tpm_infineon snd parport_pc mei_me lpc_ich mfd_core mei wmi soundcore shpchp tpm_tis parport tpm acpi_cpufreq i915 e1000e i2c_algo_bit drm_kms_helper ptp drm pps_core
 video
CPU: 1 PID: 1447 Comm: logrotate Not tainted 3.17.4-301.fc21.x86_64 #1
Hardware name: Hewlett-Packard HP Compaq 8000 Elite SFF PC/3646h, BIOS 786G7 vA1.13 10/26/2011
 0000000000000000 00000000d5882f16 ffff88003693bbb8 ffffffff8173f929
 0000000000000000 ffff88003693bbf0 ffffffff810970ad ffff8800c762b000
 ffff88003693bc40 ffff8800d8883090 ffff88000175e32c 0000000000000000
Call Trace:
 [<ffffffff8173f929>] dump_stack+0x45/0x56
 [<ffffffff810970ad>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff810971da>] warn_slowpath_null+0x1a/0x20
 [<ffffffff81228291>] drop_nlink+0x41/0x50
 [<ffffffff812a0cfb>] ext4_dec_count.isra.21+0x1b/0x30
 [<ffffffff812a671a>] ext4_rename+0x39a/0x6d0
 [<ffffffff812a6a6d>] ext4_rename2+0x1d/0x40
 [<ffffffff8121b6f3>] vfs_rename+0x4c3/0x780
 [<ffffffff8121be98>] SYSC_renameat2+0x4e8/0x560
 [<ffffffff812241a6>] ? dput+0x26/0x1b0
 [<ffffffff811c7629>] ? vma_rb_erase+0x129/0x250
 [<ffffffff811c97fc>] ? vm_munmap+0x4c/0x60
 [<ffffffff8121f23e>] SyS_rename+0x1e/0x20
 [<ffffffff81746ae9>] system_call_fastpath+0x16/0x1b