Bug 696545 - ext4 quota fs/dquot.c:814 dquot_claim_reserved_space()
Summary: ext4 quota fs/dquot.c:814 dquot_claim_reserved_space()
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.6
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Eric Sandeen
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-04-14 09:26 UTC by Wes
Modified: 2012-04-27 15:18 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-02-27 15:49:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Wes 2011-04-14 09:26:20 UTC
Description of problem:
I am getting messages in dmesg
WARNING: at fs/dquot.c:814 dquot_claim_reserved_space()

Call Trace:
 [<ffffffff8010578d>] dquot_claim_space+0xa3/0x110
 [<ffffffff88059d15>] :ext4:ext4_da_update_reserve_space+0x12d/0x194
 [<ffffffff8806ede1>] :ext4:ext4_ext_get_blocks+0x149c/0x161f
 [<ffffffff8805d165>] :ext4:ext4_da_write_end+0x23b/0x328
 [<ffffffff8000ff1f>] generic_file_buffered_write+0x20b/0x675
 [<ffffffff88059e97>] :ext4:ext4_get_blocks+0x11b/0x1cf
 [<ffffffff8805a057>] :ext4:mpage_da_map_blocks+0xaf/0x667
 [<ffffffff80048194>] pagevec_lookup_tag+0x1a/0x21
 [<ffffffff800f612a>] write_cache_pages+0x164/0x332
 [<ffffffff8805c066>] :ext4:__mpage_da_writepage+0x0/0x162
 [<ffffffff8805d8ba>] :ext4:ext4_da_writepages+0x334/0x4fc
 [<ffffffff80018314>] do_sync_write+0x0/0x104
 [<ffffffff8005ae80>] do_writepages+0x20/0x2f
 [<ffffffff8004fd9c>] __filemap_fdatawrite_range+0x50/0x5b
 [<ffffffff800504a5>] do_fsync+0x2f/0xa4
 [<ffffffff800e33f3>] __do_fsync+0x23/0x36
 [<ffffffff8005d116>] system_call+0x7e/0x83


Version-Release number of selected component (if applicable):
quota-3.13-4.el5

How reproducible:
Install centos 5.5, upgrade to 5.6,
mount ext4 partition with quota
/dev/md3 / ext4 noatime,nodiratime,defaults,usrquota 0 0
2.6.18-238.5.1.el5 #1 SMP Fri Apr 1 18:41:58 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Comment 2 Shad L. Lords 2011-05-17 20:25:02 UTC
Any update on this?

Comment 3 Jonathan Martens 2011-09-27 18:18:05 UTC
Perhaps this might be related? It seems to fix things in FC12:

https://bugzilla.redhat.com/show_bug.cgi?id=521914

Comment 4 Yuri Arabadji 2011-10-06 20:00:50 UTC
2 years to fix a bug? What's up, Redhat?

Comment 5 Ric Wheeler 2011-10-06 21:30:19 UTC
CentOS is not a Red Hat supported product. Please reproduce on a supported version of Red Hat linux with a supported file system (ext4 was only a tech preview item before RHEL5.7).

If you can reproduce on fedora or on an upstream kernel, that would also be interesting.

Thanks!

Comment 6 strovato 2011-10-17 22:07:32 UTC
I am using ext4 and RHRL5.7, a supported configuration, and can reproduce this issue.  In fact, I have been seeing this ever since converting to ext4 when 5.7 first became available.  I resorted to disabling all warnings with dmesg -n 4 on startup, but of course now I am worried about missing something important.

# dmesg

[snip]

WARNING: at fs/dquot.c:814 dquot_claim_reserved_space()

Call Trace:
 [<ffffffff80106365>] dquot_claim_space+0xa3/0x110
 [<ffffffff8805a00b>] :ext4:ext4_da_update_reserve_space+0x12e/0x195
 [<ffffffff8806f2bc>] :ext4:ext4_ext_get_blocks+0x14f9/0x167d
 [<ffffffff8014685d>] elv_merged_request+0x1e/0x26
 [<ffffffff8000c201>] __make_request+0x3e3/0x4ce
 [<ffffffff80019e3e>] __getblk+0x25/0x22c
 [<ffffffff8805a18d>] :ext4:ext4_get_blocks+0x11b/0x1cf
 [<ffffffff8805a351>] :ext4:mpage_da_map_blocks+0xb0/0x678
 [<ffffffff80047cae>] pagevec_lookup_tag+0x1a/0x21
 [<ffffffff800f6c9e>] write_cache_pages+0x164/0x334
 [<ffffffff8805c382>] :ext4:__mpage_da_writepage+0x0/0x167
 [<ffffffff80047c8b>] __pagevec_release+0x19/0x22
 [<ffffffff800de6ce>] alternate_node_alloc+0x70/0x8c
 [<ffffffff8805dbf5>] :ext4:ext4_da_writepages+0x338/0x4ff
 [<ffffffff8005a8a6>] do_writepages+0x20/0x2f
 [<ffffffff8002fa24>] __writeback_single_inode+0x1a2/0x31c
 [<ffffffff80021143>] sync_sb_inodes+0x1b7/0x271
 [<ffffffff800a2c45>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80050ce2>] writeback_inodes+0x82/0xd8
 [<ffffffff800cc364>] wb_kupdate+0xd4/0x14e
 [<ffffffff800562a9>] pdflush+0x0/0x1fb
 [<ffffffff800563fa>] pdflush+0x151/0x1fb
 [<ffffffff800cc290>] wb_kupdate+0x0/0x14e
 [<ffffffff80032722>] kthread+0xfe/0x132
 [<ffffffff8009f868>] request_module+0x0/0x14d
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff800a2c45>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032624>] kthread+0x0/0x132
 [<ffffffff8005dfa7>] child_rip+0x0/0x11

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.7 (Tikanga)

# uname -a
Linux clarus.cdeducation.org 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

# rpm -q quota
quota-3.13-5.el5

# mount |grep ext4
/dev/sda2 on / type ext4 (rw,noatime,usrjquota=aquota.user,jqfmt=vfsv0)

One time I did not mount the root filesystem with noatime, and I did not receive the warnings.  Therefore, I believe it is the combination of ext4, quota, and noatime that is triggering these.  Journaling the quota file does not have an effect.

-S

Comment 7 Eric Sandeen 2011-10-17 22:22:47 UTC
Thanks - opening a support ticket would also be helpful for tracking purposes.


The warning is:

dquot_claim_reserved_space() {
        WARN_ON(dquot->dq_dqb.dqb_rsvspace < number);

which basically means we've asked to claim more space than we have actually reserved.

Do you have any idea how to reproduce this?

The noatime thing -seems- like a red herring, since (no)atime should be unrelated to quota usage, and we have relatime on by default in any case, but perhaps, since both reporters had that option... 

0a5a9c725512461d19397490f3adf29931dca1f2 quota: Fix warning when a delayed
write happens before quota is enabled 
and
c469070aea5a0ada45a836937c776fd3083dae2b quota: manage reserved space when
quota is not active [v2]

may in fact fix it up.  I'll have to make sure they are quota-KABI safe.

Comment 8 Eric Sandeen 2011-10-17 22:27:45 UTC
(how did you get ext4 on a root filesystem?  AFAIK RHEL5 anaconda does not support that, so RHEL5 support status is probably not really there either.  That probably explains the message (IO on root happening before quota is enabled) and also explains why this hasn't been reported before (root ext4 on RHEL5 shouldn't really be possible, IIRC.)

Comment 9 Shad L. Lords 2011-10-17 22:45:17 UTC
(In reply to comment #8)
> (how did you get ext4 on a root filesystem?  AFAIK RHEL5 anaconda does not
> support that, so RHEL5 support status is probably not really there either. 
> That probably explains the message (IO on root happening before quota is
> enabled) and also explains why this hasn't been reported before (root ext4 on
> RHEL5 shouldn't really be possible, IIRC.)

root ext4 is very easy to create with a ks file.  You should also be able to create ext4 in anaconda if you pass ext4 on the boot line.

Comment 10 Eric Sandeen 2011-10-17 22:46:08 UTC
Ok, I thought that was disabled, apparently not ...

Comment 11 strovato 2011-10-17 22:47:43 UTC
I don't seem to have to do anything special to reproduce it; the warnings just
start to kick in as soon as the system starts.

The system began life with RHEL 5.0, so every filesystem including root was
formatted ext3 initially.  The upgrade to 5.7 was done via RHN not anaconda. 
The filesystem was converted to ext4 by using tune4fs to enable extents, etc.
(tune4fs -O extents,uninit_bg) followed by a manual fsck.  I have not seen any
documentation that states that ext4 is not allowed on the root filesystem?

-S

Comment 12 Eric Sandeen 2011-10-17 22:53:06 UTC
You may be right about anaconda allowing it.  However, we don't support ext3->ext4 migrations, although for a while some docs implied that we did.

(you lose a LOT of the ext4 advantages when you migrate, vs. fresh mkfs)

Well, anyway.

I would appreciate it if you could file a support ticket for this.  The 2 commits above probably take care of it.

Thanks,
-Eric

Comment 13 Ric Wheeler 2011-10-18 01:55:51 UTC
Just to second Eric, migrating in place is not supported by Red Hat or recommended. Much better just to stay on ext3 for your file system that you cannot migrate and save ext4 for the newly formatted ones.

Comment 14 strovato 2012-02-26 18:43:50 UTC
I can confirm that the issue seems to be resolved with the update to release 5.8.  Thank you.

Comment 15 Eric Sandeen 2012-02-27 15:49:41 UTC
Ok, based on comment #14, closing.  Thanks for the followup.

Comment 16 Magnus Morén 2012-04-27 10:34:53 UTC
I just noticed that I have the same problem (probably) and it is *not* fixed in 5.8.

- Fully updated 5.8 (as of today)
- uname -a
Linux xxx.yyyy.se 2.6.18-308.4.1.el5 #1 SMP Wed Mar 28 01:54:56 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
- Newly created 2TB ext4 fs (ie not converted from ext3)
- mount output
 /dev/mapper/stud1 on /home/stud1 type ext4 (rw,_netdev,usrquota,nobarrier)

(The device is a Compellent SAN volume accessed over multipath/iSCSI if this is relevant)

- removed the "noatime" and umount+mount (as of comments above)

- Large amount of errors in /var/log/messages

Apr 27 12:31:02 empok kernel: WARNING: at fs/dquot.c:814 dquot_claim_reserved_space()
Apr 27 12:31:02 empok kernel:
Apr 27 12:31:02 empok kernel: Call Trace:
Apr 27 12:31:02 empok kernel:  [<ffffffff801079bb>] dquot_claim_space+0xa3/0x110
Apr 27 12:31:02 empok kernel:  [<ffffffff88a8a2af>] :ext4:ext4_da_update_reserve_space+0x12f/0x196
Apr 27 12:31:02 empok kernel:  [<ffffffff88a9f4a8>] :ext4:ext4_ext_get_blocks+0x14e1/0x1665
Apr 27 12:31:02 empok kernel:  [<ffffffff882769a4>] :dm_mod:dm_request+0x11d/0x124
Apr 27 12:31:02 empok kernel:  [<ffffffff8001c305>] generic_make_request+0x211/0x228
Apr 27 12:31:02 empok kernel:  [<ffffffff80023301>] mempool_alloc+0x31/0xe7
Apr 27 12:31:02 empok kernel:  [<ffffffff8003309a>] submit_bio+0xe6/0xed
Apr 27 12:31:02 empok kernel:  [<ffffffff88a8a431>] :ext4:ext4_get_blocks+0x11b/0x1cf
Apr 27 12:31:02 empok kernel:  [<ffffffff88a8a5f4>] :ext4:mpage_da_map_and_submit+0xb0/0x791
Apr 27 12:31:02 empok kernel:  [<ffffffff80047c5e>] pagevec_lookup_tag+0x1a/0x21
Apr 27 12:31:02 empok kernel:  [<ffffffff800f7ce9>] write_cache_pages+0x164/0x334
Apr 27 12:31:02 empok kernel:  [<ffffffff88a8c88d>] :ext4:__mpage_da_writepage+0x0/0x154
Apr 27 12:31:02 empok kernel:  [<ffffffff88a8ddee>] :ext4:ext4_da_writepages+0x353/0x4f8
Apr 27 12:31:02 empok kernel:  [<ffffffff8005a8a3>] do_writepages+0x20/0x2f
Apr 27 12:31:02 empok kernel:  [<ffffffff8002f915>] __writeback_single_inode+0x1a2/0x31c
Apr 27 12:31:02 empok kernel:  [<ffffffff80020ff6>] sync_sb_inodes+0x1b7/0x271
Apr 27 12:31:02 empok kernel:  [<ffffffff800a328f>] keventd_create_kthread+0x0/0xc4
Apr 27 12:31:02 empok kernel:  [<ffffffff80050ce0>] writeback_inodes+0x82/0xd8
Apr 27 12:31:02 empok kernel:  [<ffffffff800ccefc>] wb_kupdate+0xf0/0x16a
Apr 27 12:31:02 empok kernel:  [<ffffffff8005626f>] pdflush+0x0/0x1fb
Apr 27 12:31:02 empok kernel:  [<ffffffff800563c0>] pdflush+0x151/0x1fb
Apr 27 12:31:02 empok kernel:  [<ffffffff800cce0c>] wb_kupdate+0x0/0x16a
Apr 27 12:31:02 empok kernel:  [<ffffffff80032632>] kthread+0xfe/0x132
Apr 27 12:31:02 empok kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Apr 27 12:31:02 empok kernel:  [<ffffffff800a328f>] keventd_create_kthread+0x0/0xc4
Apr 27 12:31:02 empok kernel:  [<ffffffff80032534>] kthread+0x0/0x132
Apr 27 12:31:02 empok kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11

Comment 17 Eric Sandeen 2012-04-27 15:18:31 UTC
Magnus, ok, we need a new bug then.  It'd also be great if you logged a support ticket.

Thanks,
-Eric


Note You need to log in before you can comment on or make changes to this bug.