Bug 481762

This is from the messages log :

Jan 27 13:21:10 node kernel: GFS: unknown option: noquotagfs_noatime
Jan 27 13:21:10 node kernel: GFS: invalid mount option(s)
Jan 27 13:21:10 node kernel: GFS: can't parse mount arguments

Is this simply a parse error somewhere?

Yes, this is a parsing error. In it's current state noatime and nodiratime mount options will cause an "Invalid Argument" error when used in conjunction with other extra options.

I've checked in a fix for this.

This seems to be working right for GFS, but is the "noquota" option supposed to work on GFS2?  "quota=(on|off)" works but "noquota" says "Invalid Argument" and the mount fails.

comment #7 is a good question... its trivial to fix, so I'd suggest opening a bug if you think it ought to accept both. I hadn't realised that the mount arguments were different.

I am adding Ric Wheeler as he is the right engineer in the right team to address this problem.

Adding Russ Doty.
I just got off the phone with Ric Wheeler and he said that this problem has been fixed.
I am not going to worry about why this has not been updated in the bugzilla but the bottom line is that this was discovered three weeks ago and has already been fixed.

Russ? Can you help get the fixes to the folks who are in need of the updated packages please?

------Mike

There seems to be some confusion around whether this problem is with GFS1 or GFS2.

Is this problem being seen with GFS1, GFS2 or both?

Are fixes needed to gfs-utils, gfs2-utils, or both?

The assigned component is gfs2-utils, but several references seem to be to gfs-utils.

I am not deeply technical but the package at the beginning of this thread is called "gfs2-utils bugs".

Denis Brækhus opened the bug: can you clarify that this is GFS2 Dennis?

It may affect gfs mounting, but the bug is actually in gfs2-utils (which gfs-utils takes advantage of).

This should be fixed in this 5.3.z package: gfs2-utils-0.1.53-1.el5_3.1

The 5.3.z should be released in the next few days.

And it will be fixed in the 5.4 release package.

Chris,

Thanks for the clarification, that helps!

Is a test version of gfs2-utils-0.1.53-1.el5_3.1 available? If so, where can SAP get it?

Also, does this include a fix for the noquota issue mentioned in comment 7? If not, can that be included before the Zstream release goes out?

Russ Doty,
Can we get this package available for the folks right away please?
I do not want to wait a few days for the zstream kit to show up in RHN.

They can get the files here:
http://people.redhat.com/cfeist/gfs2-utils-0.1.53-1.el5_3.1/

Thanks,
Chris

(In reply to comment #12)
> I am not deeply technical but the package at the beginning of this thread is
> called "gfs2-utils bugs".
> 
> Denis Brækhus opened the bug: can you clarify that this is GFS2 Dennis?

I originally opened the bug as a GFS1 bug, and as I do not have any GFS2 systems available for test at the moment I am unfortunately not able to verify whether this also applies to GFS2.

The bug as originally reported is on GFS1 systems. I do believe someone else modified the bug target component.

Regards
--
Denis

I had the same problem and subsequently installed Chris' modified gfs2-utils package which cured my immediate problem.

I have a server on which I am evaluating GFS2 and can confirm that the same problem exists on GFS2 but that the updated gfs2-utils package does NOT fix the problem.

Without looking at the source, I assume that whatever change was made to mount.gfs also needs to be applied to mount.gfs2.

I hope that this helps.

Regards,

Mick.

Regarding comment #19: Whatever problem you experienced with gfs2 cannot
be the same problem if this fix was on your system.  In fact, mount.gfs2
and mount.gfs are hard links to the same program.  Perhaps what you
experienced was bug #486168?  In either case, there's nothing wrong with
this fix, so please attach additional info to bug #486168 rather than
this one.  If it's a different issue, we may want a new bugzilla record
to track it.

(In reply to comment #20)
> Regarding comment #19: Whatever problem you experienced with gfs2 cannot
> be the same problem if this fix was on your system.  In fact, mount.gfs2
> and mount.gfs are hard links to the same program.  Perhaps what you
> experienced was bug #486168?  In either case, there's nothing wrong with
> this fix, so please attach additional info to bug #486168 rather than
> this one.  If it's a different issue, we may want a new bugzilla record
> to track it.

Hi Robert,

I hadn't noticed that they were linked.  However, I am pretty convinced that this is the same issue.  This is why:

1.  I have a GFS filesystem which I mounted with _netdev,noatime,nodiratime,noquota.

2.  Upgrade to RHEL 5.3 and I then cannot mount with the noquota and noatime/nodiratime option.

3.  Install gfs2-utils patch (above) and it now works.

4.  Convert the filesystem to GFS2 using gfs2_convert.

5.  Try the mount and it fails with "Invalid argument".

6.  /var/log/messages shows:

Feb 23 13:52:47 mtkmd01p1 kernel: GFS2: fsid=: unknown option: noquota
Feb 23 13:52:47 mtkmd01p1 kernel: GFS2: fsid=: invalid mount option(s)
Feb 23 13:52:47 mtkmd01p1 kernel: GFS2: can't parse mount arguments

This looks almost identical to the reported GFS mount bug to me....

Regards,

Mick.

Hi Mick,

I'm confused.  Are you saying that your problem is, in fact, this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=486168 ?

I'm guessing that it is, since the messages in comment #21 indicate
that the gfs2 file system is still being mounted with the "noquota"
option, which was found not to work properly in bug #486168.

Try changing your /etc/fstab to read: "quota=off" rather than "noquota"
and see if that fixes the problem.  I think it probably will work.

Versions of gfs-utils installed : 

gfs-utils-0.1.18-1.el5
gfs2-utils-0.1.53-1.el5_3.1

# cat /etc/fstab | grep /www
/dev/mapper/www	/www gfs	noquota,noatime,nodiratime		1 2
# mount /www/
/sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument

What do I have to do to make this patch work?

Regards
-- 
Denis

(In reply to comment #22)
> Hi Mick,
> I'm confused.  Are you saying that your problem is, in fact, this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=486168 ?
> I'm guessing that it is, since the messages in comment #21 indicate
> that the gfs2 file system is still being mounted with the "noquota"
> option, which was found not to work properly in bug #486168.
> Try changing your /etc/fstab to read: "quota=off" rather than "noquota"
> and see if that fixes the problem.  I think it probably will work.

Hi Robert,

My apologies for the delay.  I can confirm that quota=off does work for me.

Many thanks,

Mick.

(In reply to comment #23)
> Versions of gfs-utils installed : 
> gfs-utils-0.1.18-1.el5
> gfs2-utils-0.1.53-1.el5_3.1
> # cat /etc/fstab | grep /www
> /dev/mapper/www /www gfs noquota,noatime,nodiratime  1 2
> # mount /www/
> /sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument
> What do I have to do to make this patch work?
> Regards
> -- 
> Denis

Hi Denis,

In theory you don't have to do anything - it should work.

How did you install gfs2-utils-0.1.53-1.el5_3.1?

Regards,

Mick.

(In reply to comment #25)
> (In reply to comment #23)
> > Versions of gfs-utils installed : 
> > gfs-utils-0.1.18-1.el5
> > gfs2-utils-0.1.53-1.el5_3.1
> > # cat /etc/fstab | grep /www
> > /dev/mapper/www /www gfs noquota,noatime,nodiratime  1 2
> > # mount /www/
> > /sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument
> > What do I have to do to make this patch work?
> Hi Denis,
> In theory you don't have to do anything - it should work.
> How did you install gfs2-utils-0.1.53-1.el5_3.1?

"yum update" (the usual way).

I agree, as this is simply a fix to a tool I would expect it to resolve my issue.

Regards
--
Denis

(In reply to comment #26)
> (In reply to comment #25)
> > (In reply to comment #23)
> > > Versions of gfs-utils installed : 
> > > gfs-utils-0.1.18-1.el5
> > > gfs2-utils-0.1.53-1.el5_3.1
> > > # cat /etc/fstab | grep /www
> > > /dev/mapper/www /www gfs noquota,noatime,nodiratime  1 2
> > > # mount /www/
> > > /sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument
> > > What do I have to do to make this patch work?
> > Hi Denis,
> > In theory you don't have to do anything - it should work.
> > How did you install gfs2-utils-0.1.53-1.el5_3.1?
> "yum update" (the usual way).
> I agree, as this is simply a fix to a tool I would expect it to resolve my
> issue.
> Regards
> --
> Denis

So would I - it certainly worked for me.

I have had a similar issue in the past that I cured by removing and then reinstalling gfs-utils and gfs2-utils.  It may be worth a try.

If that doesn't work then I am sure that one of the experts on here will be able to help.

Sorry I can't be of more help.

Regards,

Mick.

(In reply to comment #23)
> Versions of gfs-utils installed : 
> 
> gfs-utils-0.1.18-1.el5
> gfs2-utils-0.1.53-1.el5_3.1
> 
> # cat /etc/fstab | grep /www
> /dev/mapper/www /www gfs noquota,noatime,nodiratime  1 2
> # mount /www/
> /sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument

I just tried the same test case with these exact packages.
What error messages do you see in the dmesg output?
Can you run "rpm -V gfs-utils gfs2-utils" to verify that the packages are installed correctly?

Hi Denis,

You also need to have the latest gfs kernel package on your system
as well.  I recreated your same problem and it was solved by installing
the kmod-gfs-0.1.31-3.el5 package.  If you check "dmesg" you'll likely
see messages like these:

Feb 25 08:45:11 roth-01 kernel: GFS: can't parse mount arguments
Feb 25 08:45:51 roth-01 kernel: GFS: unknown option: gfs_noatime

Bob Peterson

(In reply to comment #29)
> You also need to have the latest gfs kernel package on your system
> as well.  I recreated your same problem and it was solved by installing
> the kmod-gfs-0.1.31-3.el5 package.  If you check "dmesg" you'll likely
> see messages like these:
> 
> Feb 25 08:45:11 roth-01 kernel: GFS: can't parse mount arguments
> Feb 25 08:45:51 roth-01 kernel: GFS: unknown option: gfs_noatime

Hi Bob,

You are correct, these errors (similar to the ones I originally reported) do appear.

Well, these packages are installed, and I was pretty sure the box was rebooted after that too, but upon inspection it does seem I have various versions of kmod-gfs installed : 

kmod-gfs-0.1.23-5.el5_2.2
kmod-gfs-0.1.23-5.el5_2.4
kmod-gfs-0.1.31-3.el5
kmod-gfs2-1.92-1.1.el5
kmod-gfs2-1.92-1.1.el5_2.2
gfs-utils-0.1.18-1.el5
gfs2-utils-0.1.53-1.el5_3.1

Any way to check what version of the module is currently loaded?
uname -a does confirm that the box has recently been rebooted with the "2.6.18-128.1.1.el5" kernel. And from what I gather, the only kmod-gfs to suppy a 2.6.18-128 version of the module is kmod-gfs-0.1.31-3.el5, but this could be irrelevant for all I know.

Is the multiple versions of the package a potential problem here?

I am not currently able to reboot the node at will, but I can schedule a planned reboot later on if necessary.

Regards
-- 
Denis

Modinfo gives:

# modinfo gfs
filename:       /lib/modules/2.6.18-128.1.1.el5/weak-updates/gfs/gfs.ko
license:        GPL
author:         Red Hat, Inc.
description:    Global File System 0.1.23-5.el5_2.4
srcversion:     B66A601FEACF3031BDBA2EA
depends:        gfs2
vermagic:       2.6.18-92.1.13.el5 SMP mod_unload gcc-4.1

Regards
-- 
Denis

Whatever was fixed in gfs2-utils-0.1.53-1.el5_3.1, it wasn't the main underlying breakage:

I have the following package versions:
kernel-2.6.18-92.1.22.el5
gfs-utils-0.1.18-1
gfs2-utils-0.1.53-1.el5_3.1
kmod-gfs-0.1.23-5.el5_2.4

And when trying to mount a GFS (not GFS2!) volume, I get the following:
# mount /dev/drbd1 /mnt/newroot -o noatime
GFS: unknown option: gfs_noatime                                                
GFS: invalid mount option(s)                                                    
GFS: can't parse mount arguments                                                
/sbin/mount.gfs: error mounting /dev/drbd1 on /mnt/newroot: Invalid argument  

nodiratime also seems broken:
# mount /dev/drbd1 /mnt/newroot -o nodiratime
GFS: unknown option: gfs_noatime                                                
GFS: invalid mount option(s)                                                    
GFS: can't parse mount arguments                                                
/sbin/mount.gfs: error mounting /dev/drbd1 on /mnt/newroot: Invalid argument  

Can anyone confirm what was the last package version combination that worked correctly?

nodiratime is not supported.

The issue that you've discovered with mounting gfs is related to a previous problem that was "fixed" some time back. Here are the details:

atime is not a per-superblock option, its a per mount point option. Its implemented in the VFS and not the filesystem. As a result of that, the VFS takes the noatime (and nodiratime if you specify it) options out of the list it supplies to the filesystem before it passes the information to the fileystems mount routine. This was changed in the VFS a number of kernel versions ago, resulting in the noatime options being silently ignored.

The problem is that gfs was historically written to assume that it would take care of the noatime option (the VFS didn't provide a suitable mode for gfs, but it does now in the form of relatime) on a per superblock basis.

Since it was going to be too invasive a change to make atime a per mount option in gfs, the userland mount helper was updated to change the noatime option into gfs_noatime which was then passed to the filesystem (since the VFS wouldn't try to interpret this new option). It looks like what has happened here is that you've got new userland tools which do that, combined with an old kernel which doesn't understand the option. I'd suggest upgrading your kmod/kernel.

In gfs2, we use the VFS's implementation of the noatime options, so it works just like any other filesystem (as a per mount point option).

OK, that's fair enough, but in that case there is still a packaging bug in that rpm packages gfs-utils-0.1.18-1 and/or gfs2-utils-0.1.53-1.el5_3.1 should have a dependency on kernel >= 2.6.18-128 and/or kmod-gfs >= 0.1.31 (I'm assuming these are the minimum versions required to keep noatime parameter functional) since there is clearly a compatibility issue, minor as it may be. Package/version dependency requirements are, after all, the big advantage of RPM over just using tar balls. It may be splitting hairs, but I think the point is valid.

Hi,

I'm using RHEL 5.2 (GFS v1), with some packages from 5.3 trying to solve this issue but nothing until now:

[root@csd-poa-cla-08 ~]# rpm -qa | grep gfs
kmod-gfs2-1.92-1.1.el5_2.2
kmod-gfs-0.1.31-3.el5
kmod-gfs-0.1.23-5.el5_2.4
gfs2-utils-0.1.53-1.el5_3.1
gfs-utils-0.1.18-1.el5

[root@csd-poa-cla-08 ~]# uname -a
Linux csd-poa-cla-08.ig.com.br 2.6.18-128.1.1.el5 #1 SMP Mon Jan 26 13:58:24 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

[root@csd-poa-cla-08 ~]# mount /dev/csd/data /shared -o noatime
/sbin/mount.gfs: error mounting /dev/mapper/csd-data on /shared: Invalid argument

[root@csd-poa-cla-08 ~]# dmesg 
GFS 0.1.23-5.el5_2.4 (built Oct  6 2008 17:45:52) installed
GFS: unknown option: gfs_noatime
GFS: invalid mount option(s)
GFS: can't parse mount arguments

What can I do? Thanks

This issue is not fixed for me, I have all updates applied as of today 2009-04-15, and rebooted both clusternodes after last updates.

Packages installed : 
# rpm -qa "*gfs*"
gfs2-utils-0.1.53-1.el5_3.2
gfs-utils-0.1.18-1.el5
kmod-gfs2-1.92-1.1.el5_2.2
kmod-gfs2-1.92-1.1.el5
kmod-gfs-0.1.23-5.el5_2.4
kmod-gfs-0.1.23-5.el5_2.2
kmod-gfs-0.1.31-3.el5

Module loads :
# dmesg | grep GFS 
GFS2 Overlay (built May 29 2008 16:48:00) installed
GFS 0.1.23-5.el5_2.4 (built Oct  6 2008 17:45:52) installed
GFS: fsid=cluster_X:www_gfs.1: jid=1: Trying to acquire journal lock...
GFS: fsid=cluster_X:www_gfs.1: jid=1: Looking at journal...
GFS: fsid=cluster_X:www_gfs.1: jid=1: Done

Normal fstab only has "noquota" option, trying to use additional options fails :

# mount -o noatime,nodiratime,noquota,remount /www/
/sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument
# mount -o noatime,noquota,remount /www/
/sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument
# mount -o nodiratime,noquota,remount /www/
/sbin/mount.gfs: error mounting /dev/mapper/www on /www: Invalid argument

Kernel and modinfo :

# uname -a 
Linux node 2.6.18-128.1.6.el5 #1 SMP Tue Mar 24 12:05:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

# modinfo gfs
filename:       /lib/modules/2.6.18-128.1.6.el5/weak-updates/gfs/gfs.ko
license:        GPL
author:         Red Hat, Inc.
description:    Global File System 0.1.23-5.el5_2.4
srcversion:     B66A601FEACF3031BDBA2EA
depends:        gfs2
vermagic:       2.6.18-92.1.13.el5 SMP mod_unload gcc-4.1


--
Denis

Can anybody please give a statement what should be done to support the noatime feature of current RHEL GFS filesystems?

As I understood it if we leave it out it will - though not "supported" anymore - degrade performance drastically.

So the question still is, what should be done with filesystems being mounted with noatime/noquota option?

So far as the noquota option goes, all gfs versions support that. Early gfs2 didn't support it, but quota=off is equivalent (and indeed the documented way of doing that). We have now made gfs2 support noquota in addition to quota=off and that will appear in RHEL 5.4.

Prior to RHEL5, noatime worked as expected on GFS. From the start of RHEL5, due to the new way in which the VFS was dealing with the noatime option, it was not being passed on to gfs or gfs2.

The gfs2 fix was to to use the VFS's atime code rather than its own and that was fixed in RHEL 5.3: https://bugzilla.redhat.com/show_bug.cgi?id=462579

At the same time a fix was made to gfs to use the gfs_noatime parameter which would be passed by the VFS to the filesystem. When mount.gfs (since the fix) is given notime as a mount argument it translates it into gfs_noatime and passes that option to the kernel. The filesystem then knows that it should not to atime and this avoids the VFS eating the option on the way.

As a result, you need to upgrade the kernel and the userland tools in step to make this feature work. Older RHEL5 kernels are not able to turn off atime.

Can you please explicitly list as of which versions of the RHEL 5.3 kernel and other required packages in the dependency tree of this issue the gfs_noatime is implemented and functioning correctly? The initial 5.3 release version of the packages didn't seem to work with noatime.

I looked for the packages that have the right patches in gfs2-utils and kmod-gfs and here they are:

gfs2-utils-0.1.53-1.el5_3.1
gfs-kmod-0.1.28 (in RHEL 5.3 and beyond)

Please ensure that the right gfs.ko is loaded.

Ok, right, the noatime option with kmod-gfs-0.1-31 is working as expected. Great!

Looks like the weak-modules called by the rpms implicitly didn't work. After deinstalling all kmods and reinstalling it works now as expected.

But still the question resides if it is reasonable to stay with the noatime option.

Steve, if I got it right the noatime even bridged over to the gfs-mod does nothing?

Yes, using noatime is recommended unless you application dictates otherwise.

I'm not sure what you mean by "bridged over"?

Forget about the bridged over. 

I'll keep in mind that noatime is still recommended and that is perfectly ok with me.

Thanks.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1337.html