Bug 1486132

Created attachment 1319326 [details]
backup.groovy

Hello Miroslav,

This is an illegal clone, zstream are cloned from ystream.
Please add RHEL 5 y stream flag in the bug.

@Oneata Sorry, I don't know what's the correct process and not sure what RHEL 5 y stream flag is. Could you set the correct flags instead of me?

Thanks,
Mirek

(In reply to Miroslav Novak from comment #3)
> @Oneata Sorry, I don't know what's the correct process and not sure what
> RHEL 5 y stream flag is. Could you set the correct flags instead of me?
> 
> Thanks,
> Mirek

Hi There,
Sorry, typo there, RHEL 7.5 as y stream flag

Its my understanding that there are two NFS clients reading and writing to
the same file on a server. Both clients and server are RHEL 7.4

When the backup client is active, it holds a lock on the 2ed byte 
of the file and then monitors the 1st byte of the file to see 
when to shutdown.

When the live client sees the 2ed byte is locked, it writes to
the 1st byte of the file, telling the backup to shutdown.

In 7.3 the backup client was seeing the live client's write 
and in 7.4 it does not. Basically you got lucky with 7.3 because
the only way to guarantee synchronization between two NFS client 
reads and writes to the same file is through file locking.

So the solution here is to use the same locking synchronization on 
the 1st byte as done on the 2ed byte. 

Have the live client lock the 1st byte, write the byte then 
unlock the file which will flush the byte back to the server. 
Then have the backup client lock the 1st, read the byte then 
unlock the file. 

If that does not work, there is bug in NFS...

Thanks Steve! I've tried your suggestion and it works. I believe this bugzilla can be closed as not a bug.

I totally overlooked the "sync" options on those mount points.

If the NFS mount points on both clients involved were mounted with "sync", then the extra locking should be unnecessary.  From nfs(5):

 "If the sync option is specified on a mount point, any system call that
  writes data to files on that mount point causes that data to be flushed
  to the server before the system call returns control to user space."

If both clients are mounting with "sync", then the extra locking should make no difference, but Comment 7 says it did make a difference.

So, I'm stumped.  Are we positive about those mount options?

> Are we positive about those mount options?

Yes. Mount options are the same on both of the NFS clients.

(In reply to J. Bruce Fields from comment #11)
> If both clients are mounting with "sync", then the extra locking should make
> no difference, but Comment 7 says it did make a difference.

Apologies, my mistake: sync doesn't turn off attribute or data caching.  So, writes on one client are being flushed to the server before the system call returns, but reads on the other client may still rely on cache.  You'd need "noac", not just sync, to eliminate the need for locking.

So, the behavior looks to me as expected, and I think we can close this as NOTABUG.

Apologies again for the confusion.

There is "noac" option as well. The goal was to avoid any caching or asynchronous behaviour in mount to keep state between NFS clients in sync.

(In reply to Miroslav Novak from comment #15)
> There is "noac" option as well. The goal was to avoid any caching or
> asynchronous behaviour in mount to keep state between NFS clients in sync.

Argh, OK, thanks.

What's the server?  (And, if Linux, what filesystem are you exporting, and what are the mount options?)

Server is RHEL 7.4. Exported file system is xfs:
/dev/mapper/system-hornetq--nfs41                                                       xfs        10G   81M   10G   1% /mnt/system-hornetq/client-nfs41

But i'm not sure if this can have effect as I could see the same behaviour when NFSv4 server was RHEL 6.9 and exported file system was ext4. NFS clients were RHEL 7.4 with the above mount options.

(In reply to Miroslav Novak from comment #17)
> Server is RHEL 7.4. Exported file system is xfs:
> /dev/mapper/system-hornetq--nfs41                                           
> xfs        10G   81M   10G   1% /mnt/system-hornetq/client-nfs41
> 
> But i'm not sure if this can have effect as I could see the same behaviour
> when NFSv4 server was RHEL 6.9 and exported file system was ext4. NFS
> clients were RHEL 7.4 with the above mount options.

I wondered if there might be a problem with NFSv4 change attribute resolution.  Even with "noac", the client probably still depends on the change attribute to know whether the file has changed, and older xfs will use a change attribute that's actually based on a last-modified time (ctime actually) with resolution about 1ms, which can lead the client to miss changes that occur in very quick succession--I'm not clear whether this test case might be doing that.

Could you check with "xfs_db -r -c version /dev/mapper/system-hornetq--nfs41" that V5 is set?  Also if possible it'd be useful to know whether MS_I_VERSION is actually set on that mount.... See https://bugzilla.redhat.com/show_bug.cgi?id=1466981#c7 for an example of doing this with crash, I don't know another way.

Sorry, I cannot execute xfs_db in our QA lab as don't have 'root' rights. I there another way how to check it.

I've asked one of the admins to check it, here is output for: "xfs_db -r -c version /dev/mapper/system-hornetq--nfs41":

versionnum [0xb4b4+0x8a] = V4,NLINK,DIRV2,ATTR,ALIGN,LOGV2,EXTFLG,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT

> I wondered if there might be a problem with NFSv4 change attribute
> resolution.  Even with "noac", the client probably still depends on the
> change attribute to know whether the file has changed, and older xfs will
> use a change attribute that's actually based on a last-modified time (ctime
> actually) with resolution about 1ms, which can lead the client to miss
> changes that occur in very quick succession--I'm not clear whether this test
> case might be doing that.

I think this is not the case as backup.groovy is reading 1st byte of server.lock file periodically (in while cycle). So if it misses update once then it would get it next update.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.