Description of problem:

I have a 4+2 disperse volume with ctime on, and export a dir from nfs-ganesha,

storage.ctime: on
features.utime: on

When I copy a local file to nfs client, stat shows bad ctime for the file.

# stat /mnt/nfs/test*
  File: ‘/mnt/nfs/test1.sh’
  Size: 166             Blocks: 4          IO Block: 1048576 regular file
Device: 27h/39d Inode: 10744358902712050257  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-08-05 09:49:00.000000000 +0800
Modify: 2019-08-05 09:49:00.000000000 +0800
Change: 2061-07-23 21:54:08.000000000 +0800
 Birth: -
  File: ‘/mnt/nfs/test2.sh’
  Size: 214             Blocks: 4          IO Block: 1048576 regular file
Device: 27h/39d Inode: 12073556847735387788  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-08-05 09:49:00.000000000 +0800
Modify: 2019-08-05 09:49:00.000000000 +0800
Change: 2061-07-23 21:54:08.000000000 +0800
 Birth: -

# ps a
342188 pts/0    D+     0:00 cp -i test1.sh test2.sh /mnt/nfs/

# gdb glusterfsd
(gdb) p *stbuf
$1 = {ia_flags = 0, ia_ino = 0, ia_dev = 0, ia_rdev = 0, ia_size = 0,
  ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_blksize = 0, ia_blocks = 0,
  ia_atime = 174138658, ia_mtime = 2889352448, ia_ctime = 0, ia_btime = 0,
  ia_atime_nsec = 0, ia_mtime_nsec = 0, ia_ctime_nsec = 0, ia_btime_nsec = 0,
  ia_attributes = 0, ia_attributes_mask = 0,
  ia_gfid = '\000' <repeats 15 times>, ia_type = IA_INVAL, ia_prot = {
    suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {
      read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, group = {
      read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {
      read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}}

It is caused by nfs client create the copied file as EXCLUSIVE mode which
set a verifier, the verifier is set to file's atime and mtime.

nfs client set the verifier as, 

        if (flags & O_EXCL) {
                data->arg.create.createmode  = NFS3_CREATE_EXCLUSIVE;
                data->arg.create.verifier[0] = cpu_to_be32(jiffies);
                data->arg.create.verifier[1] = cpu_to_be32(current->pid);
        }
the verifier[0] is set to file's atime, and verifier[1] is set to mtime.

But utime at storage/posix set the mtime to ctime too at setattr and set ctime to a earlier time is not allowed. 

        /* Earlier, mdata was updated only if the existing time is less
         * than the time to be updated. This would fail the scenarios
         * where mtime can be set to any time using the syscall. Hence
         * just updating without comparison. But the ctime is not
         * allowed to changed to older date.
         */

The following codes is used to find those PIDs which may cause a bad ctime for a copied file.

==========================================================================
#include <stdio.h>
#include <stdlib.h>

int swap_endian(int val){
        val = ((val << 8)&0xFF00FF00) | ((val >> 8)&0x00FF00FF);
        return (val << 16)|(val >> 16);
}

// time of 2020/01/01 0:0:0
#define TO2020 1577808000

int main(int argc, char **argv)
{
        unsigned int i = 0, val = 0;
        for (i = 0; i < 500000; i++) {
                val = swap_endian(i);
                if (val > TO2020)
                        printf("%u %u\n", i, val);
        }
        return 0;
}