Created attachment 157293 [details]
A test program

Miloslav, is this happening on x86_64?

If so, I think we need this kernel patch in -stable:

http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=272a3713bb9e302e0455c894c41180a482d2c8a3

(and maybe the next one committed as well)

This is i686 (Pentium M) on Intel 855.

Happens on FC6 (i686) too.
Not sure this is really a bug...

Hello,

I'm reviewing this bug as part of the kernel bug triage project, an attempt to
isolate current bugs in the fedora kernel.

http://fedoraproject.org/wiki/KernelBugTriage

I am CC'ing myself to this bug and will try and assist you in resolving it if I can.

There hasn't been much activity on this bug for a while. Could you tell me if
you are still having problems with the latest kernel and if you consider this a
bug as Chuck has inidicated it may not be?

If the problem no longer exists then please close this bug or I'll do so in a
few days if there is no additional information lodged.

Cheers
Chris

Retested on rawhide - kernel-2.6.23-0.202.rc8.fc8.

And I definitely consider this a bug - not only as a principle that the OS
should never return results inconsistent with causality, but also because this
can cause mlocate to miss directory changes.

I would say this is a candidate for taking upstream if you're willing to do so
Miloslav. I am happy to file a bug report in the kernel.org bugzilla for you if
you have an account that I can CC you into there as well and are willing to
respond to the technical queries.

Whats the latest on this - do we wish to file upstream? FWIW I still see this on
F8 and if its causing issues for mlocate I'd like to see it resolved...

Running the test program on XFS, which supports nanosecond resolution, and with
a small change to print the full time from gettimeofday() I get:

1199906098.363249311
1199906098.364405
1199906098.364244817

Looks like the mtime is 0.000160183 seconds behind what is returned by
gettimeofday()... and when the time is very near to an exact second boundary the
rounding causes a 1-second difference.

Created attachment 291225 [details]
simple test

An even simpler test (from the filesystem perspective) is to just rename a
file, and look at mtime.  Same behavior.

Ugh, ignore me, of course rename doesn't change mtime on the file being renamed :/

Happens on uniprocessor and happens when booting with "nohz=off".

Also happens in a (valid this time) simple case like this:

gettimeofday()
creat("file")
stat("file")

stat mtime sometimes precedes time from gettimeofday.

gettimeofday uses xtime directly, mtime is set via things like CURRENT_TIME,
which uses xtime_cached....

This message is a reminder that Fedora 7 is nearing the end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 7. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '7'.

Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 7's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 7 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug. If you are unable to change the version, please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. If possible, it is recommended that you try the newest available Fedora distribution to see if your bug still exists.

Please read the Release Notes for the newest Fedora distribution to make sure it will meet your needs:
http://docs.fedoraproject.org/release-notes/

The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping

This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Reproduced on kernel-2.6.29.4-167.fc11.x86_64

Seems to be fixed in 2.6.31

Reproduced again on kernel-2.6.31.9-174.fc12.x86_64 , took less than 5 seconds.

1262975533.999822647
1262975534
1262975533.999822647

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource 
hpet acpi_pm 
[mitr@kulicka t]$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet

hi,
I probably found the cause and send "startup patch" to lkml.
You can get some info in here http://lkml.org/lkml/2010/7/2/59

wbr,
jirka

(In reply to comment #19)
> hi,
> I probably found the cause and send "startup patch" to lkml.
> You can get some info in here http://lkml.org/lkml/2010/7/2/59

The test program in the above should resolve it, but will cause the file timestamping to be costly, causing performance issues.

The issue here is that the vsyscall-time() implementation is using gettimeofday instead of returning the xtime value passed to the update_vsyscall().

Fixing it can be more complex, as on older kernels, the xtime value passed will not match current_kernel_time either. So the fix I'm suggesting will only work with 2.6.34+ kernels that include the remove xtime_cache patch (commit 6a867a395558a7f882d041783e4cdea6744ca2bf)

Possible patch to fix this on mainline is on lkml: http://lkml.org/lkml/2010/7/6/300

This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Reproduced again on 2.6.34.7-61.fc13.x86_64.

This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Reproduced again on kernel-2.6.38.6-27.fc15.x86_64.

Miloslav, Looking at the test case orignally provided, I think the behavior is expected. Filesystem timestamps are course-grained, and have "tick" or HZ resolution.

Where as gettimeofday provides more fine-grained time (as fine grained as the underlying clocksource hardware). Interleaving these two different-granular timestamps will give apparent inconsistencies. 

If you replaced gettimeofday() with time() or even clock_gettime(CLOCK_REALTIME_COARSE, ...) you should not see any inconsistencies.

thanks
-john

(In reply to comment #26)
> Miloslav, Looking at the test case orignally provided, I think the behavior is
> expected. Filesystem timestamps are course-grained, and have "tick" or HZ
> resolution.
> 
> Where as gettimeofday provides more fine-grained time (as fine grained as the
> underlying clocksource hardware). Interleaving these two different-granular
> timestamps will give apparent inconsistencies. 

No, there is only one actual "real" time.  Filesystem timestamps may have lower granularity, but because they are supposed to be always rounded down, there should be no apparent contradictions with causality.

This behavior is also a direct contradiction with POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html 4.8, last paragraph.  For example, in comment #0, after gettimeofday() returns 1182183921.something, we know for sure that at the time of rename() V >= 1182183921.0.  Per the same POSIX paragraph, the resolution is at most one second, so the timestamp after rename() should be at least 1182183921, not 1182183920.

> If you replaced gettimeofday() with time() or even
> clock_gettime(CLOCK_REALTIME_COARSE, ...) you should not see any
> inconsistencies.
Sure, but that's only because the internal implementation of FS timestamps works based on CLOCK_REALTIME_COARSE, not because there is any underlying logic in the relation.


I realize that fixing this bug would have unwelcome performance implications; but that doesn't really make the behavior correct.

The current behavior has been present in Linux for as long as we have had sub-tick resolution timestamps.  While issues have cropped up in the past (ie: time() not matching filesystem timestamps) to my knowledge those issues have been resolved.

If you want to see nanosecond granular filesystem timestamps (currently it is nanosecond resolution, but tick granular), you're welcome to try to push for such a change in the kernel, but past attempts to do so have not been successful (due to the performance implications).

(In reply to comment #28)
> If you want to see nanosecond granular filesystem timestamps (currently it is
> nanosecond resolution, but tick granular),
Just to clarify, I'm perfectly happy with one-second granularity filesystem timestamps - as long as it really is 1 second, not 2 seconds.  (And of course mlocate can compensate for this.)

I'm thinking this might be a good thing to bring to the linux-fsdevel list, since it's not Fedora-specific?

-Eric

(In reply to comment #29)
> (In reply to comment #28)
> > If you want to see nanosecond granular filesystem timestamps (currently it is
> > nanosecond resolution, but tick granular),
> Just to clarify, I'm perfectly happy with one-second granularity filesystem
> timestamps - as long as it really is 1 second, not 2 seconds.  (And of course
> mlocate can compensate for this.)

Right, but that requires the ticks to be perfectly second aligned (which isn't really possible, due to both NTP adjustments and irq latencies), or you need to do the slower fine-grained gettime when creating filesystem timestamps.

But yea, I agree with Eric that fsdevel is where this conversation would be more productive.

Fedora 15 has reached it's end of life as of June 26, 2012.  As a result, we will not be fixing any remaining bugs found in Fedora 15.

In the event that you have upgraded to a newer release and the bug you reported is still present, please reopen the bug and set the version field to the newest release you have encountered the issue with.  Before doing so, please ensure you are testing the latest kernel update in that release and attach any new and relevant information you may have gathered.

Thank you for taking the time to file a report.  We hope newer versions of Fedora suit your needs.

Reproduced again on kernel-3.4.4-5.fc17.x86_64.

This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Reproduced in kernel-3.9.6-200.fc18.x86_64.

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 18 kernel bugs.

Fedora 18 has now been rebased to 3.11.4-101.fc18.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 19, and are still experiencing this issue, please change the version to Fedora 19.

If you experience different issues, please open a new bug report for those.

Reproduced with kernel-3.11.4-201.fc19.x86_64.

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.12.6-200.fc19.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 20, and are still experiencing this issue, please change the version to Fedora 20.

If you experience different issues, please open a new bug report for those.

Reproduced with kernel-3.12.6-200.fc19.x86_64.

Please send this bug report to the upstream fs list, linux-fsdevel.org

I feel bad punting this way, but obviously I'm not getting to this bug - there's hopefully more bandwidth on the list than I have.

Thanks,
-Eric

I've also had fun with this a few years ago: bug #636255