Bug 244697 - Time goes backward - gettimeofday() vs. rename()
Time goes backward - gettimeofday() vs. rename()
Status: ASSIGNED
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
rawhide
x86_64 Linux
low Severity medium
: ---
: ---
Assigned To: Kernel Maintainer List
Brian Brock
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-06-18 12:30 EDT by Miloslav Trmač
Modified: 2015-09-25 16:07 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-07-11 13:49:34 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
A test program (939 bytes, text/plain)
2007-06-18 12:30 EDT, Miloslav Trmač
no flags Details
simple test (862 bytes, text/x-csrc)
2008-01-09 22:06 EST, Eric Sandeen
no flags Details

  None (edit)
Description Miloslav Trmač 2007-06-18 12:30:42 EDT
Occasionally, a rename() fails to update st_mtime of the parent directory to a
"current" time, as observed by a previous gettimeofday () call.

Version-Release number of selected component (if applicable):
kernel-2.6.21-1.3228.fc7

Steps to Reproduce:
1. compile the attachment
2 [details]. while :; do ./test-mtime || break; done

Actual results:
The loop terminates after one or two seconds, with the last iteration showing
something like:
  1182183920.000000000
  1182183921
  1182183920.000000000

Expected results:
The loop runs endlessly.

Additional info:
selected parts of dmesg:
  Time: tsc clocksource has been installed.
  PCI: Setting latency timer of device 0000:00:1e.0 to 64
  Time: acpi_pm clocksource has been installed.
  Clocksource tsc unstable (delta = -105614334 ns)
# cat /sys/devices/system/clocksource/clocksource0/available_clocksource 
acpi_pm pit jiffies tsc 
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource 
acpi_pm
Comment 1 Miloslav Trmač 2007-06-18 12:30:42 EDT
Created attachment 157293 [details]
A test program
Comment 2 Chuck Ebbert 2007-06-18 21:11:35 EDT
Miloslav, is this happening on x86_64?

If so, I think we need this kernel patch in -stable:

http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=272a3713bb9e302e0455c894c41180a482d2c8a3

(and maybe the next one committed as well)
Comment 3 Miloslav Trmač 2007-06-18 21:23:10 EDT
This is i686 (Pentium M) on Intel 855.
Comment 4 Chuck Ebbert 2007-06-19 18:04:58 EDT
Happens on FC6 (i686) too.
Not sure this is really a bug...
Comment 5 Christopher Brown 2007-09-17 07:17:29 EDT
Hello,

I'm reviewing this bug as part of the kernel bug triage project, an attempt to
isolate current bugs in the fedora kernel.

http://fedoraproject.org/wiki/KernelBugTriage

I am CC'ing myself to this bug and will try and assist you in resolving it if I can.

There hasn't been much activity on this bug for a while. Could you tell me if
you are still having problems with the latest kernel and if you consider this a
bug as Chuck has inidicated it may not be?

If the problem no longer exists then please close this bug or I'll do so in a
few days if there is no additional information lodged.

Cheers
Chris
Comment 6 Miloslav Trmač 2007-09-25 18:09:37 EDT
Retested on rawhide - kernel-2.6.23-0.202.rc8.fc8.

And I definitely consider this a bug - not only as a principle that the OS
should never return results inconsistent with causality, but also because this
can cause mlocate to miss directory changes.
Comment 7 Christopher Brown 2007-09-26 16:50:13 EDT
I would say this is a candidate for taking upstream if you're willing to do so
Miloslav. I am happy to file a bug report in the kernel.org bugzilla for you if
you have an account that I can CC you into there as well and are willing to
respond to the technical queries.
Comment 8 Christopher Brown 2008-01-09 09:53:02 EST
Whats the latest on this - do we wish to file upstream? FWIW I still see this on
F8 and if its causing issues for mlocate I'd like to see it resolved...
Comment 9 Chuck Ebbert 2008-01-09 14:20:54 EST
Running the test program on XFS, which supports nanosecond resolution, and with
a small change to print the full time from gettimeofday() I get:

1199906098.363249311
1199906098.364405
1199906098.364244817

Looks like the mtime is 0.000160183 seconds behind what is returned by
gettimeofday()... and when the time is very near to an exact second boundary the
rounding causes a 1-second difference.
Comment 10 Eric Sandeen 2008-01-09 22:06:08 EST
Created attachment 291225 [details]
simple test

An even simpler test (from the filesystem perspective) is to just rename a
file, and look at mtime.  Same behavior.
Comment 11 Eric Sandeen 2008-01-09 22:24:42 EST
Ugh, ignore me, of course rename doesn't change mtime on the file being renamed :/
Comment 12 Chuck Ebbert 2008-01-10 15:00:56 EST
Happens on uniprocessor and happens when booting with "nohz=off".
Comment 13 Eric Sandeen 2008-01-10 15:14:15 EST
Also happens in a (valid this time) simple case like this:

gettimeofday()
creat("file")
stat("file")

stat mtime sometimes precedes time from gettimeofday.

gettimeofday uses xtime directly, mtime is set via things like CURRENT_TIME,
which uses xtime_cached....
Comment 14 Bug Zapper 2008-05-14 09:09:49 EDT
This message is a reminder that Fedora 7 is nearing the end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 7. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '7'.

Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 7's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 7 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug. If you are unable to change the version, please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. If possible, it is recommended that you try the newest available Fedora distribution to see if your bug still exists.

Please read the Release Notes for the newest Fedora distribution to make sure it will meet your needs:
http://docs.fedoraproject.org/release-notes/

The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 15 Bug Zapper 2009-06-09 18:39:46 EDT
This message is a reminder that Fedora 9 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 9.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '9'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 9's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 9 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 16 Miloslav Trmač 2009-06-10 08:50:30 EDT
Reproduced on kernel-2.6.29.4-167.fc11.x86_64
Comment 17 Chuck Ebbert 2009-10-06 11:44:31 EDT
Seems to be fixed in 2.6.31
Comment 18 Miloslav Trmač 2010-01-08 13:33:28 EST
Reproduced again on kernel-2.6.31.9-174.fc12.x86_64 , took less than 5 seconds.

1262975533.999822647
1262975534
1262975533.999822647

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource 
hpet acpi_pm 
[mitr@kulicka t]$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hpet
Comment 19 Jiri Olsa 2010-07-02 03:55:45 EDT
hi,
I probably found the cause and send "startup patch" to lkml.
You can get some info in here http://lkml.org/lkml/2010/7/2/59

wbr,
jirka
Comment 20 john stultz 2010-07-06 18:08:24 EDT
(In reply to comment #19)
> hi,
> I probably found the cause and send "startup patch" to lkml.
> You can get some info in here http://lkml.org/lkml/2010/7/2/59

The test program in the above should resolve it, but will cause the file timestamping to be costly, causing performance issues.

The issue here is that the vsyscall-time() implementation is using gettimeofday instead of returning the xtime value passed to the update_vsyscall().

Fixing it can be more complex, as on older kernels, the xtime value passed will not match current_kernel_time either. So the fix I'm suggesting will only work with 2.6.34+ kernels that include the remove xtime_cache patch (commit 6a867a395558a7f882d041783e4cdea6744ca2bf)
Comment 21 john stultz 2010-07-06 21:02:09 EDT
Possible patch to fix this on mainline is on lkml: http://lkml.org/lkml/2010/7/6/300
Comment 22 Bug Zapper 2010-11-04 08:09:53 EDT
This message is a reminder that Fedora 12 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 12.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '12'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 12's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 12 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 23 Miloslav Trmač 2010-11-04 08:23:56 EDT
Reproduced again on 2.6.34.7-61.fc13.x86_64.
Comment 24 Bug Zapper 2011-06-02 14:41:24 EDT
This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 13 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 25 Miloslav Trmač 2011-06-03 08:37:50 EDT
Reproduced again on kernel-2.6.38.6-27.fc15.x86_64.
Comment 26 john stultz 2011-06-03 14:37:37 EDT
Miloslav, Looking at the test case orignally provided, I think the behavior is expected. Filesystem timestamps are course-grained, and have "tick" or HZ resolution.

Where as gettimeofday provides more fine-grained time (as fine grained as the underlying clocksource hardware). Interleaving these two different-granular timestamps will give apparent inconsistencies. 

If you replaced gettimeofday() with time() or even clock_gettime(CLOCK_REALTIME_COARSE, ...) you should not see any inconsistencies.

thanks
-john
Comment 27 Miloslav Trmač 2011-06-07 14:50:14 EDT
(In reply to comment #26)
> Miloslav, Looking at the test case orignally provided, I think the behavior is
> expected. Filesystem timestamps are course-grained, and have "tick" or HZ
> resolution.
> 
> Where as gettimeofday provides more fine-grained time (as fine grained as the
> underlying clocksource hardware). Interleaving these two different-granular
> timestamps will give apparent inconsistencies. 

No, there is only one actual "real" time.  Filesystem timestamps may have lower granularity, but because they are supposed to be always rounded down, there should be no apparent contradictions with causality.

This behavior is also a direct contradiction with POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html 4.8, last paragraph.  For example, in comment #0, after gettimeofday() returns 1182183921.something, we know for sure that at the time of rename() V >= 1182183921.0.  Per the same POSIX paragraph, the resolution is at most one second, so the timestamp after rename() should be at least 1182183921, not 1182183920.

> If you replaced gettimeofday() with time() or even
> clock_gettime(CLOCK_REALTIME_COARSE, ...) you should not see any
> inconsistencies.
Sure, but that's only because the internal implementation of FS timestamps works based on CLOCK_REALTIME_COARSE, not because there is any underlying logic in the relation.


I realize that fixing this bug would have unwelcome performance implications; but that doesn't really make the behavior correct.
Comment 28 john stultz 2011-06-07 15:22:04 EDT
The current behavior has been present in Linux for as long as we have had sub-tick resolution timestamps.  While issues have cropped up in the past (ie: time() not matching filesystem timestamps) to my knowledge those issues have been resolved.

If you want to see nanosecond granular filesystem timestamps (currently it is nanosecond resolution, but tick granular), you're welcome to try to push for such a change in the kernel, but past attempts to do so have not been successful (due to the performance implications).
Comment 29 Miloslav Trmač 2011-06-07 15:38:43 EDT
(In reply to comment #28)
> If you want to see nanosecond granular filesystem timestamps (currently it is
> nanosecond resolution, but tick granular),
Just to clarify, I'm perfectly happy with one-second granularity filesystem timestamps - as long as it really is 1 second, not 2 seconds.  (And of course mlocate can compensate for this.)
Comment 30 Eric Sandeen 2011-06-07 15:47:28 EDT
I'm thinking this might be a good thing to bring to the linux-fsdevel list, since it's not Fedora-specific?

-Eric
Comment 31 john stultz 2011-06-07 16:33:38 EDT
(In reply to comment #29)
> (In reply to comment #28)
> > If you want to see nanosecond granular filesystem timestamps (currently it is
> > nanosecond resolution, but tick granular),
> Just to clarify, I'm perfectly happy with one-second granularity filesystem
> timestamps - as long as it really is 1 second, not 2 seconds.  (And of course
> mlocate can compensate for this.)

Right, but that requires the ticks to be perfectly second aligned (which isn't really possible, due to both NTP adjustments and irq latencies), or you need to do the slower fine-grained gettime when creating filesystem timestamps.

But yea, I agree with Eric that fsdevel is where this conversation would be more productive.
Comment 32 Josh Boyer 2012-07-11 13:49:34 EDT
Fedora 15 has reached it's end of life as of June 26, 2012.  As a result, we will not be fixing any remaining bugs found in Fedora 15.

In the event that you have upgraded to a newer release and the bug you reported is still present, please reopen the bug and set the version field to the newest release you have encountered the issue with.  Before doing so, please ensure you are testing the latest kernel update in that release and attach any new and relevant information you may have gathered.

Thank you for taking the time to file a report.  We hope newer versions of Fedora suit your needs.
Comment 33 Miloslav Trmač 2012-07-11 13:55:23 EDT
Reproduced again on kernel-3.4.4-5.fc17.x86_64.
Comment 34 Fedora End Of Life 2013-07-04 02:48:34 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 35 Miloslav Trmač 2013-07-04 09:15:47 EDT
Reproduced in kernel-3.9.6-200.fc18.x86_64.
Comment 36 Justin M. Forbes 2013-10-18 16:59:57 EDT
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 18 kernel bugs.

Fedora 18 has now been rebased to 3.11.4-101.fc18.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 19, and are still experiencing this issue, please change the version to Fedora 19.

If you experience different issues, please open a new bug report for those.
Comment 37 Miloslav Trmač 2013-10-21 14:41:42 EDT
Reproduced with kernel-3.11.4-201.fc19.x86_64.
Comment 38 Justin M. Forbes 2014-01-03 17:04:20 EST
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.12.6-200.fc19.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 20, and are still experiencing this issue, please change the version to Fedora 20.

If you experience different issues, please open a new bug report for those.
Comment 39 Miloslav Trmač 2014-01-06 12:57:15 EST
Reproduced with kernel-3.12.6-200.fc19.x86_64.
Comment 40 Eric Sandeen 2014-01-07 15:21:39 EST
Please send this bug report to the upstream fs list, linux-fsdevel@vger.kernel.org

I feel bad punting this way, but obviously I'm not getting to this bug - there's hopefully more bandwidth on the list than I have.

Thanks,
-Eric
Comment 41 Stefan Ring 2015-09-25 16:07:36 EDT
I've also had fun with this a few years ago: bug #636255

Note You need to log in before you can comment on or make changes to this bug.