Bug 981574 - mount failed with access deny, and mountd log: "qword_eol: fflush failed: errno 22 (Invalid argument)"
mount failed with access deny, and mountd log: "qword_eol: fflush failed: err...
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: nfs-utils (Show other bugs)
19
All Linux
unspecified Severity high
: ---
: ---
Assigned To: J. Bruce Fields
Fedora Extras Quality Assurance
: Reopened, TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-05 03:32 EDT by Yin.JianHong
Modified: 2015-02-18 08:58 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-02-18 08:58:26 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Fix parse-exports problems with exportfs's test_export (3.43 KB, patch)
2013-09-13 18:17 EDT, J. Bruce Fields
no flags Details | Diff
Fix exportfs not to pass down -1 uid/gid on test_export (976 bytes, patch)
2013-09-13 18:21 EDT, J. Bruce Fields
no flags Details | Diff

  None (edit)
Description Yin.JianHong 2013-07-05 03:32:06 EDT
Description of problem:
Do a simple local mount(/tmp *(sync)) failed. access deny.
the mountd write the follow log:
"Jul  5 15:18:43 dhcp12-241 rpc.mountd[11641]: authenticated mount request from 10.66.12.241:922 for /tmp (/tmp)
Jul  5 15:18:43 dhcp12-241 rpc.mountd[11641]: qword_eol: fflush failed: errno 22 (Invalid argument)
Jul  5 15:18:43 dhcp12-241 rpc.mountd[11641]: Cannot export /tmp, possibly unsupported filesystem or fsid= required"

Version-Release number of selected component (if applicable):
nfs-utils-1.2.8-2.0.fc19.x86_64
3.9.6-301.fc19.x86_64

How reproducible:
In my host, 100%

Steps to Reproduce:
1. echo "/tmp *(sync)"  >/etc/exports
2. service nfs restart
3. mount $ip:/tmp  $mp

Actual results:
# mount 10.66.xx.xxx:/tmp  /media/
mount.nfs: access denied by server while mounting 10.66.xx.xxx:/tmp

Expected results:
mount OK.

Additional info:
the follow method can not resolve the problem:
'every fopen() on the procfs files in question is followed by
    a call to setvbuf(), using a per-file dedicated buffer of
    RPC_CHAN_BUF_SIZE length.'
Comment 1 Yin.JianHong 2013-07-05 03:34:54 EDT
mount at remote host(new installed 6.0, 6.4)  get same result.
Comment 2 sumanth k 2013-09-09 12:36:37 EDT
There is a workaround for this problem . 

If we include fsid in the /etc/exports , then this problem doesnt occur. mount will be successfull.

man page of exports specifies the following : 
NFS needs to be able to identify each filesystem that it exports.  Normally it will use a UUID for the filesystem (if the filesystem has such a thing) or the device number of the device holding the filesystem (if the filesystem is stored on the device).

As not all filesystems are stored on devices, and not all filesystems have UUIDs, it is sometimes necessary to explicitly tell NFS how to identify a  filesystem. This  is  done with the fsid= option. 

For  NFSv4,  there  is  a  distinguished  filesystem which is the root of all exported filesystem.  This is specified with fsid=root or fsid=0 both of which mean exactly the same thing.


Tested the above and works fine when we include fsid option.
Comment 3 Yin.JianHong 2013-09-09 21:52:25 EDT
(In reply to sumanth k from comment #2)
> There is a workaround for this problem . 
> 
> If we include fsid in the /etc/exports , then this problem doesnt occur.
> mount will be successfull.
> 
> man page of exports specifies the following : 
> NFS needs to be able to identify each filesystem that it exports.  Normally
> it will use a UUID for the filesystem (if the filesystem has such a thing)
> or the device number of the device holding the filesystem (if the filesystem
> is stored on the device).
> 
> As not all filesystems are stored on devices, and not all filesystems have
> UUIDs, it is sometimes necessary to explicitly tell NFS how to identify a 
> filesystem. This  is  done with the fsid= option. 
> 
> For  NFSv4,  there  is  a  distinguished  filesystem which is the root of
> all exported filesystem.  This is specified with fsid=root or fsid=0 both of
> which mean exactly the same thing.
> 
> 
> Tested the above and works fine when we include fsid option.

But, it works always fine, in latest RHEL7; not need fsid= specify
Comment 4 J. Bruce Fields 2013-09-11 11:10:23 EDT
(In reply to Yin.JianHong from comment #3)
> But, it works always fine, in latest RHEL7; not need fsid= specify

Probably Fedora is putting /tmp on tmpfs and RHEL7 is putting it on a "real" filesystem.

Out of curiosity: could you strace mountd?  I'd like to see what it's actually trying to pass down to the kernel that's resulting in the einval.  So, run:

  strace -p $(pidof rpc.mountd) -s4096 -e trace=open,close,read,write

then try your test case.
Comment 5 Yin.JianHong 2013-09-11 19:53:38 EDT
(In reply to J. Bruce Fields from comment #4)
> (In reply to Yin.JianHong from comment #3)
> > But, it works always fine, in latest RHEL7; not need fsid= specify
> 
> Probably Fedora is putting /tmp on tmpfs and RHEL7 is putting it on a "real"
> filesystem.
got it, what's the difference in tmpfs and real fs?

> 
> Out of curiosity: could you strace mountd?  I'd like to see what it's
> actually trying to pass down to the kernel that's resulting in the einval. 
> So, run:
> 
>   strace -p $(pidof rpc.mountd) -s4096 -e trace=open,close,read,write
> 
> then try your test case.
console2:
[root@dhcp12-241 autofs]# mount 127.0.0.1:/tmp   /opt/
mount.nfs: access denied by server while mounting 127.0.0.1:/tmp

console1:
[root@dhcp12-241 ~]# strace -p $(pidof rpc.mountd) -s4096 -e trace=open,close,read,write
Process 22780 attached
open("/var/lib/nfs/etab", O_RDONLY)     = 12
close(12)                               = 0
open("/proc/net/rpc/auth.unix.ip/channel", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
write(12, "nfsd 127.0.0.1 1378945012 * \n", 29) = 29
close(12)                               = 0
open("/proc/net/rpc/nfsd.export/channel", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
open("/sys/dev/block/0:31", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
write(12, "* /tmp 1378945012 1060 65534 65534 0 secinfo 1 1 1060 \n", 55) = -1 EINVAL (Invalid argument)
close(12)                               = 0
----------------------------------------------
And now cur version:
[root@dhcp12-241 autofs]# rpm -q nfs-utils kernel
nfs-utils-1.2.8-3.0.fc19.x86_64
kernel-3.9.9-302.fc19.x86_64
kernel-3.10.3-300.fc19.x86_64
kernel-3.10.4-300.fc19.x86_64
[root@dhcp12-241 autofs]# uname -r
3.10.4-300.fc19.x86_64
Comment 6 Yin.JianHong 2013-09-11 19:56:53 EDT
BTW:  mount with nfsv3, same result:
console2:
[root@dhcp12-241 autofs]# mount -o vers=3 127.0.0.1:/tmp   /opt/
mount.nfs: access denied by server while mounting 127.0.0.1:/tmp

console1:
[root@dhcp12-241 ~]# strace -p $(pidof rpc.mountd) -s4096 -e trace=open,close,read,write
Process 22780 attached
open("/var/lib/nfs/etab", O_RDONLY)     = 12
close(12)                               = 0
open("/proc/net/rpc/auth.unix.ip/channel", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
write(12, "nfsd 127.0.0.1 1378945502 * \n", 29) = 29
close(12)                               = 0
open("/proc/net/rpc/nfsd.export/channel", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
open("/sys/dev/block/0:31", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
close(13)                               = 0
write(12, "* /tmp 1378945502 1060 65534 65534 0 secinfo 1 1 1060 \n", 55) = -1 EINVAL (Invalid argument)
close(12)                               = 0
Comment 7 J. Bruce Fields 2013-09-12 09:44:27 EDT
The -EINVAL is almost certainly returned from the following check in fs/nfsd/export.c:check_export():

        /* There are two requirements on a filesystem to be exportable.
         * 1:  We must be able to identify the filesystem from a number.
         *       either a device number (so FS_REQUIRES_DEV needed)
         *       or an FSID number (so NFSEXP_FSID or ->uuid is needed).
         * 2:  We must be able to find an inode from a filehandle.
         *       This means that s_export_op must be set.
         */
        if (!(inode->i_sb->s_type->fs_flags & FS_REQUIRES_DEV) &&
            !(*flags & NFSEXP_FSID) &&
            uuid == NULL) {
                dprintk("exp_export: export of non-dev fs without fsid\n");
                return -EINVAL;
        }

That looks right to me.  So I believe the existing behavior is correct.

I'd recommend using some filesystem other than /tmp for your test.
Comment 8 Yin.JianHong 2013-09-12 19:43:11 EDT
(In reply to J. Bruce Fields from comment #7)
> The -EINVAL is almost certainly returned from the following check in
> fs/nfsd/export.c:check_export():
> 
>         /* There are two requirements on a filesystem to be exportable.
>          * 1:  We must be able to identify the filesystem from a number.
>          *       either a device number (so FS_REQUIRES_DEV needed)
>          *       or an FSID number (so NFSEXP_FSID or ->uuid is needed).
>          * 2:  We must be able to find an inode from a filehandle.
>          *       This means that s_export_op must be set.
>          */
>         if (!(inode->i_sb->s_type->fs_flags & FS_REQUIRES_DEV) &&
>             !(*flags & NFSEXP_FSID) &&
>             uuid == NULL) {
>                 dprintk("exp_export: export of non-dev fs without fsid\n");
>                 return -EINVAL;
>         }
> 
> That looks right to me.  So I believe the existing behavior is correct.
> 
> I'd recommend using some filesystem other than /tmp for your test.

Ok, I understand; Thank you for your explanation.

I think that:  can we probe this issue at parseing the exports item?
I mean when I start the nfs, I could get a error/warning:
   error: '$dir' export of non-dev fs without fsid, you need specify fsid=<ID>
Comment 9 Yin.JianHong 2013-09-13 08:23:40 EDT
I have a debug, and review the code of exportfs,
find that: this is a bug of exportfs(in fact is kernel):

static int test_export(char *path, int with_fsid)
{
        char buf[1024];
        int fd, n;

        sprintf(buf, "-test-client- %s 3 %d -1 -1 0\n",
                path,
                with_fsid ? NFSEXP_FSID : 0);
        fd = open("/proc/net/rpc/nfsd.export/channel", O_WRONLY);
        if (fd < 0)
                return 0;
        n = write(fd, buf, strlen(buf));
        close(fd);
        if (n < 0)
                return 0;
        return 1;
}

1. if the path is an tmpfs(no uuid), and with_fsid==0,
this function test_export still return 1 (write success). that's wrong.

2. and another problem: validate_export() just generate log and return void.
We think that validate_export() should return an int value to indicate whether the path can export. And in export_all(), should not export the path if invalid;

static void
export_all(int verbose)
{
        nfs_export      *exp;
        int             i;

        for (i = 0; i < MCL_MAXTYPES; i++) {
                for (exp = exportlist[i].p_head; exp; exp = exp->m_next) {
                        if (verbose)
                                printf("exporting %s:%s\n",
                                       exp->m_client->m_hostname,
                                       exp->m_export.e_path);
                        exp->m_xtabent = 1;
                        exp->m_mayexport = 1;
                        exp->m_changed = 1;
                        exp->m_warned = 0;
                        validate_export(exp);
                }
        }
}


static void <<<- should return int value to indicate whether the path can export
validate_export(nfs_export *exp)
{
Comment 10 J. Bruce Fields 2013-09-13 10:20:05 EDT
(In reply to Yin.JianHong from comment #8)
> I think that:  can we probe this issue at parseing the exports item?
> I mean when I start the nfs, I could get a error/warning:
>    error: '$dir' export of non-dev fs without fsid, you need specify
> fsid=<ID>

Excellent point, I agree, thanks for persisting.

> 1. if the path is an tmpfs(no uuid), and with_fsid==0,
> this function test_export still return 1 (write success). that's wrong.

I can reproduce that here.  I'll investigate.

I think I agree on the export_all() and validate_export() behavior too.  I'm not working on that for now (patches welcomed).
Comment 11 J. Bruce Fields 2013-09-13 18:17:10 EDT
Created attachment 797546 [details]
Fix parse-exports problems with exportfs's test_export

The user-namespace patches added some code which silently rejects exports with -1 uid & gid before they even get to the test_export checks.  The attached patches fix the problem for me, and I now get a helpful error from exportfs.
Comment 12 J. Bruce Fields 2013-09-13 18:21:11 EDT
Created attachment 797547 [details]
Fix exportfs not to pass down -1 uid/gid on test_export

We can also fix this with an nfs-utils patch.  Either the kernel patch or this nfs-utils patch is sufficient to solve the problem, but we should probably do both.
Comment 15 Fedora End Of Life 2015-01-09 17:10:52 EST
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 16 Fedora End Of Life 2015-02-18 08:58:26 EST
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Note You need to log in before you can comment on or make changes to this bug.