Bug 2363149 - Moving or copying a file from NFS to any other filesystem generates an error
Summary: Moving or copying a file from NFS to any other filesystem generates an error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: coreutils
Version: 42
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Lukáš Zaoral
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-04-30 17:42 UTC by Thomas Clark
Modified: 2025-05-21 02:17 UTC (History)
12 users (show)

Fixed In Version: coreutils-9.7-2.fc43 coreutils-9.6-3.fc42
Clone Of:
Environment:
Last Closed: 2025-05-19 10:58:32 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
strace of mv (6.01 KB, text/plain)
2025-05-10 23:56 UTC, Thomas Clark
no flags Details
current coreutils source tarball (5.79 MB, application/x-gtar)
2025-05-11 01:44 UTC, Paul Eggert
no flags Details
mv trace from coreutils tarball (6.06 KB, text/plain)
2025-05-11 02:24 UTC, Thomas Clark
no flags Details

Description Thomas Clark 2025-04-30 17:42:40 UTC
Moving or copying a file FROM an NFS filesystem TO any other filesystem generates an error "preserving permissions for <target directory>/<filename>: Operation not supported."  Moving or copying succeeds, and the basic permissions are copied.  The error happens whether or not the target file has extended attributes.  Copying from a non-NFS filesystem to an NFS filesystem does not generate the error. Copying from an NFS filesystem to a different NFS filesystem does not generate the error. This behavior began with Fedora 42.

Reproducible: Always

Steps to Reproduce:
1.Create a file on an NFS filesystem
2.Move the file to a non-NFS filesystem
3.
Actual Results:
preserving permissions for <filename>: Operation not supported.

Expected Results:
No error

Comment 1 Lukáš Zaoral 2025-05-05 13:35:16 UTC
Hello Thomas,
this is expected behaviour.  I agree that the error message is a bit annoying,
but it is, technically, correct.  Is it not possible to preserve NFSv4 ACLs
(the "system.nfs4_acl" xattr) when copying files to a non-NFS filesystem, as witnessed by
the following strace invocation:

# #  strace -e 'trace=/.*xattr.*' cp --preserve=mode /mnt/nfs4/file .
flistxattr(3, NULL, 0)                  = 16
flistxattr(3, "system.nfs4_acl\0", 16)  = 16
fgetxattr(3, "system.nfs4_acl", NULL, 0) = 80
fgetxattr(3, "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80) = 80
fsetxattr(4, "system.nfs4_acl", "\0\0\0\3\0\0\0\0\0\0\0\0\0\26\1\207\0\0\0\6OWNER@\0\0\0\0\0", 80, 0) = -1 EOPNOTSUPP (Operation not supported)
cp: preserving permissions for ‘file’: Operation not supported
+++ exited with 1 +++

FYI, copying of NFSv4 ACLs is now tried unconditionally since coreutils 9.6 and was introduced in the following gnulib commit: https://github.com/coreutils/gnulib/commit/47947855dda53fd12bbae2a0fccecb2280577a60#diff-2c141e9e9f39b6f17da0fdc2945a52753dd4bd7aee2ec9145878675a5edde7d7R50-R55

Regards,
Lukas

Comment 2 Thomas Clark 2025-05-05 16:02:46 UTC
Lukas, thanks for your response. However, the error is misleading and incorrect.  The copy operation does correctly preserve basic attributes, such as owner and permissions, but the error is generated nevertheless, and it states "operation not supported" even though there was no actual error.  The error should be generated only if a copy operation failed to preserve attributes. Producing this error when there was no actual error causes the operator to think that the copy operation failed when it did not. This IS a bug.

Comment 3 Lukáš Zaoral 2025-05-06 08:34:12 UTC
Thanks for the response, Thomas.

> The error should be generated only if a copy operation failed to preserve attributes.

Which is exactly your case!  According to the info page of cp(1), copying with
--preserve=mode will preserve all permissions *including any ACL and xattr permissions*,
which also includes the "system.nfs4_acl" xattr:

> ‘mode’
>          Preserve attributes relevant to access permissions, including
>          file mode bits and (if possible) access control lists (ACLs).
>          ACL preservation is system-dependent, and ACLs are not
>          necessarily translated when the source and destination are on
>          file systems with different ACL formats (e.g., NFSv4 versus
>          POSIX formats).

Since coreutils 9.6 improved the support for handling of this NFSv4-only permission xattr
and the target file is on an NFSv4 share, cp will try to preserve it as well.  Unfortunately,
non-NFS filesystems do to support this xattr.  Therefore, the respective syscall fails with ENOTSUP.

If you have any other questions or if my explanation was unclear, please, let me know.

Comment 4 Pádraig Brady 2025-05-06 10:01:27 UTC
I've not looked into the details of this TBH,
but NFSv4 ACLs can be "trivial" in which case
we probably should not warn.

As I said I've not looked into the details of this case,
but it would be good to double check that we do not
warn if there are only trivial NFSv4 ACLs.
The gnulib code already has acl_nfs4_nontrivial()
to determine this.

Comment 5 Thomas Clark 2025-05-06 15:30:47 UTC
It's actually not my case. The files I am copying have only trivial attributes, and they transfer correctly, yet I get an error suggesting that the copy operation failed. Again, the error should happen ONLY if attributes ACTUALLY, not theoretically, failed to transfer.  I am running these mv commands in batch jobs, and I am now getting strings of incorrect error messages every time one of them runs.  The only way I can figure out to stop the errors is to throw away all of the error messages, but then I will have no way of knowing whether a copy actually failed. Please take another look at this.

Comment 6 Lukáš Zaoral 2025-05-06 17:05:29 UTC
Thank you both, Pádraig and Thomas!

I've knocked together a dirty gnulib patch which should attempt to copy acl
xargs only if they are non-trivial.  Thomas, if you'd be interested, I could
provide you with a test RPM with this patch applied before a proper fix is
landed upstream.

diff --git a/lib/qcopy-acl.c b/lib/qcopy-acl.c
index ad7966152a..f746c0c666 100644
--- a/lib/qcopy-acl.c
+++ b/lib/qcopy-acl.c
@@ -27,6 +27,7 @@

 # include <attr/libattr.h>
 # include <string.h>
+# include <dirent.h>

 # if HAVE_LINUX_XATTR_H
 #  include <linux/xattr.h>
@@ -85,11 +86,14 @@ qcopy_acl (const char *src_name, int source_desc, const char *dst_name,
      (i.e. posix <-> nfs4) but we can't do it anyway, so for now, we don't care
      Functions attr_copy_* return 0 in case we copied something OR nothing
      to copy */
-  if (ret == 0)
+  struct aclinfo ai;
+  if (ret == 0 && file_has_aclinfo (src_name, &ai, DT_UNKNOWN) == 1) {
+    aclinfo_free (&ai);
     ret = source_desc <= 0 || dest_desc <= 0
       ? attr_copy_file (src_name, dst_name, is_attr_permissions, NULL)
       : attr_copy_fd (src_name, source_desc, dst_name, dest_desc,
                       is_attr_permissions, NULL);
+  }
 #else
   /* no XATTR, so we proceed the old dusty way */
   struct permission_context ctx;

Comment 7 Thomas Clark 2025-05-06 18:05:48 UTC
Thanks. I would be happy to test it.

Comment 8 Lukáš Zaoral 2025-05-07 11:24:38 UTC
Thomas, here is the test build: https://koji.fedoraproject.org/koji/taskinfo?taskID=132420703

Note that this patch was not tested thoroughly and is not suitable for upstream submission in the current form.

Comment 9 Thomas Clark 2025-05-07 12:57:33 UTC
The patched version handles copying correctly. It produces no error when attributes are preserved.  When ACLs are lost, it correctly produces an error. It's now thoroughly tested. :)

Comment 10 Paul Eggert 2025-05-10 05:29:29 UTC
(In reply to Lukáš Zaoral from comment #8)
> Note that this patch was not tested thoroughly and is not suitable for
> upstream submission in the current form.

I looked at the patch and found a potential memory leak (aclinfo_free wasn't always called) and more importantly, the patch causes extra system calls to be executed even for normal platforms that don't have this NFSv4 problem. I wrote what I hope is a better patch, and published it here:

https://bugs.gnu.org/78328#11

This patch is installed in Coreutils upstream. I hope some NFSv4 users who experience the bug can test the fix, as I didn't reproduce the bug.

Comment 11 Thomas Clark 2025-05-10 17:40:23 UTC
Paul Eggert, I tested the new patch (but only for the permission error) and it does not produce an error when permissions were not lost. However, it also does not produce an error when ACLs ARE lost.  To prove this, I created a file on an NFS host and added ACLs. I then went to an NFS client and moved the file to an ext4 filesystem. The ACLs were lost, and no error was produced. I also noticed today that, although I was using them interchangeably, the 'mv' and 'cp' commands do not produce errors the same way. That could explain why you were not seeing errors on your NFS filesystem.  Try it with the 'mv' command. It always produces the permission error.

Comment 12 Paul Eggert 2025-05-10 19:35:18 UTC
(In reply to Thomas Clark from comment #11)
> I created a file on an NFS host and added ACLs. I then went to an NFS client and moved
> the file to an ext4 filesystem. The ACLs were lost, and no error was
> produced.

Thanks for looking into it. What is the strace output for the failing command? E.g.:

  cd /nfs/file/system
  touch nfsfile
  setfacl -m u:bin:rw nfsfile
  LC_ALL=C strace -o mv.tr mv nfsfile /ext4/file/system

What system calls do you see near the end of mv.tr that indicate what mv is doing incorrectly?

>the 'mv' and 'cp' commands do not produce errors the same
> way. That could explain why you were not seeing errors on your NFS
> filesystem.

I can't reproduce it with mv either. My NFS server (which I don't control) doesn't let me create files with nontrivial ACLs.

Comment 13 Thomas Clark 2025-05-10 23:56:38 UTC
Created attachment 2089315 [details]
strace of mv

Comment 14 Thomas Clark 2025-05-10 23:58:38 UTC
My NFS server doesn't let me create nontrivial ACLs on a client, but I can do it on the underlying filesystem on the NFS server.  I did so, and then executed a mv to an xfs filesystem. It completed without an error and the ACLs were, of course, lost. The strace file is attached above.

Comment 15 Paul Eggert 2025-05-11 01:44:51 UTC
Created attachment 2089320 [details]
current coreutils source tarball

This is a copy of the bleeding-edge coreutils tarball. To build, do this:

xz -d <coreutils-9.7.13-6bd4.tar.xz | tar xf -
cd coreutils-9.7.13-6bd4.tar.xz
./configure
make

Comment 16 Paul Eggert 2025-05-11 01:46:38 UTC
The crucial part of that strace looks like this:

  renameat2(AT_FDCWD, "nfsfile", AT_FDCWD, "/var/tmp/nfsfile", RENAME_NOREPLACE) = -1 EXDEV (Invalid cross-device link)
  openat(AT_FDCWD, "/var/tmp/nfsfile", O_RDONLY|O_PATH|O_DIRECTORY) = -1 ENOENT (No such file or directory)
  newfstatat(AT_FDCWD, "nfsfile", {st_mode=S_IFREG|0664, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
  newfstatat(AT_FDCWD, "/var/tmp/nfsfile", 0x7ffe8f4f7c30, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
  unlinkat(AT_FDCWD, "/var/tmp/nfsfile", 0) = -1 ENOENT (No such file or directory)
  openat(AT_FDCWD, "nfsfile", O_RDONLY|O_NOFOLLOW) = 3
  fstat(3, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
  openat(AT_FDCWD, "/var/tmp/nfsfile", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
  ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = -1 EXDEV (Invalid cross-device link)
  fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
  fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
  uname({sysname="Linux", nodename="fedora.ferree-clark.org", ...}) = 0
  copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = -1 EXDEV (Invalid cross-device link)
  mmap(NULL, 1056768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5ff9d81000
  read(3, "", 1048576)                    = 0
  utimensat(4, NULL, [{tv_sec=1746920974, tv_nsec=966910501} /* 2025-05-10T16:49:34.966910501-0700 */, {tv_sec=1746920974, tv_nsec=966910501} /* 2025-05-10T16:49:34.966910501-0700 */], 0) = 0

At this point I would expect calls to flistxattr, to see whether the source file has extended attributes. Can you investigate why this is not happening on your platform?

What are the values of HAVE_LINUX_XATTR_H and HAVE_LISTXATTR in lib/config.h? They should both be 1.

Also, please build from the latest coreutils sources so that we can be sure we're on the same page. I have attached a current tarball, and you can build from that. If you use any special build procedure (rather than ./configure; make) please let us know.

Thanks.

Comment 17 Thomas Clark 2025-05-11 02:23:04 UTC
I rebuilt from your tarfile. The only change I made was config --prefix=/usr.  It still gives no error upon moving a file from an NFS filesystem to an XFS filesystem where ACLs were lost. I will attach the mv.tr file.

Comment 18 Thomas Clark 2025-05-11 02:24:58 UTC
Created attachment 2089322 [details]
mv trace from coreutils tarball

Comment 19 Paul Eggert 2025-05-11 04:49:13 UTC
Thanks for the new trace. It shows the following:

  ...
  read(4, "", 1048576)                    = 0
  utimensat(5, NULL, [{tv_sec=1746929774, tv_nsec=747761911} /* 2025-05-10T19:16:14.747761911-0700 */, {tv_sec=1746929774, tv_nsec=747761911} /* 2025-05-10T19:16:14.747761911-0700 */], 0) = 0
  fchown(5, 2153, 1500)                   = 0
  fchmod(5, 0100664)                      = 0
  close(5)                                = 0

That is, there is no attempt to call flistxattr or any similar function. This suggests that the development environment doesn't have the xattr libraries installed, which means cp and mv won't try to copy extended attributes.

To test that theory, what do the following shell commands output?

  grep XATTR lib/config.h
  ldd src/mv

On my platform (RHEL 9.5) they output the following:

  $ grep XATTR lib/config.h
  #define HAVE_LINUX_XATTR_H 1
  #define HAVE_LISTXATTR 1
  /* #undef PTHREAD_MUTEXATTR_ROBUST_UNIMPLEMENTED */
  #define USE_XATTR 1
  $ ldd src/mv
	linux-vdso.so.1 (0x00007fff1c3f8000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f47ba7c2000)
	libacl.so.1 => /lib64/libacl.so.1 (0x00007f47ba7b7000)
	libattr.so.1 => /lib64/libattr.so.1 (0x00007f47ba7af000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f47ba400000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f47ba713000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f47ba804000)

Also, if you look at config.log, do you see lines like the following? If not, what do you see?

  configure:8430: checking for linux/xattr.h
  configure:8430: gcc -c -g -O2  conftest.c >&5
  configure:8430: $? = 0
  configure:8430: result: yes
  ...
  configure:10757: checking for listxattr
  configure:10757: gcc -o conftest -g -O2   conftest.c  >&5
  configure:10757: $? = 0
  configure:10757: result: yes

Comment 20 Thomas Clark 2025-05-11 13:38:20 UTC
I am testing on a clean installation of Fedora 42.

grep XATTR lib/config.h
#define HAVE_LINUX_XATTR_H 1
#define HAVE_LISTXATTR 1
/* #undef PTHREAD_MUTEXATTR_ROBUST_UNIMPLEMENTED */
/* #undef USE_XATTR */

ldd src/mv
	linux-vdso.so.1 (0x00007febfd58c000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007febfd549000)
	libc.so.6 => /lib64/libc.so.6 (0x00007febfd357000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007febfd2ac000)
	/lib64/ld-linux-x86-64.so.2 (0x00007febfd58e000)


configure:8430: checking for linux/xattr.h
configure:8430: gcc -c -g -O2  conftest.c >&5
configure:8430: $? = 0
configure:8430: result: yes

configure:10757: checking for listxattr
configure:10757: gcc -o conftest -g -O2   conftest.c  >&5
configure:10757: $? = 0
configure:10757: result: yes

I also found
configure:71327: WARNING: libattr development library was not found or not usable.
configure:71329: WARNING: GNU coreutils will be built without xattr support.configure:11621: WARNING: libacl development library was not found or not usable.
configure:11623: WARNING: GNU coreutils will be built without ACL support.

and
configure:11621: WARNING: libacl development library was not found or not usable.
configure:11623: WARNING: GNU coreutils will be built without ACL support.

After installing the missing development libraries and rebuilding, the mv command now properly displays an error when it causes the loss of ACLs and does display an error when ACLs were not present.

However, the cp command does not display an error even when ACLs are lost. Is that intended behavior? Perhaps it is, in a sense, true that ACLs are not lost on a cp command because they are still on the original file, but it still seems to me that the two commands should behave identically with regard to lost attributes.

Comment 21 Paul Eggert 2025-05-11 15:10:32 UTC
(In reply to Thomas Clark from comment #20)

> After installing the missing development libraries and rebuilding, the mv
> command now properly displays an error when it causes the loss of ACLs and
> does display an error when ACLs were not present.

Thanks for checking.

> However, the cp command does not display an error even when ACLs are lost.
> Is that intended behavior?

Yes, unless one uses an option like 'cp -p' or 'cp --preserve=xattr'. Have you tried those options?

Comment 22 Thomas Clark 2025-05-11 16:18:15 UTC
Sorry, I should have figured out the cp flags. The cp -p command works as intended. Thank you for your help! I assume your patch is already headed to the release version?

Comment 23 Paul Eggert 2025-05-11 16:23:08 UTC
(In reply to Thomas Clark from comment #22)
> I assume your patch is already headed to
> the release version?

Yes, the fix should appear in the next upstream coreutils release.

Comment 24 Pádraig Brady 2025-05-12 10:16:42 UTC
Reopening this, as @lzaoral it would be good to get this fix included with the Fedora 9.6 package.
BTW @eggert if confirmed it's a >= 9.6 issue, we need to tweak the version mentioned in upstream NEWS.

thanks everyone for looking into this

Comment 25 Fedora Update System 2025-05-19 09:19:06 UTC
FEDORA-2025-62fee954be (coreutils-9.7-2.fc43) has been submitted as an update to Fedora 43.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-62fee954be

Comment 26 Fedora Update System 2025-05-19 09:24:11 UTC
FEDORA-2025-7f2e17c13e (coreutils-9.6-3.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-7f2e17c13e

Comment 27 Fedora Update System 2025-05-19 10:58:32 UTC
FEDORA-2025-62fee954be (coreutils-9.7-2.fc43) has been pushed to the Fedora 43 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 28 Fedora Update System 2025-05-20 02:34:06 UTC
FEDORA-2025-7f2e17c13e has been pushed to the Fedora 42 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-7f2e17c13e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-7f2e17c13e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 29 Fedora Update System 2025-05-21 02:17:31 UTC
FEDORA-2025-7f2e17c13e (coreutils-9.6-3.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.