Here is a problem most likely due to ext3, although it could be related to find and/or apache. There seems to be a related bug for memory mapped files, but I suspect this is not the case here -- and the corresponding descriptions (there seems to be more than one report) are either vague or confidential. And in that case the problem seem to involve st_ctime and st_mtime. A computer is running apache 2.2, providing access to a few files and directories after a basic successful login. When access is provided to a single file (for download) the access time stamp seems not to be updated. At least trying eg the shell command find $HTTP_DIR -type f -atime -2 -exec ls -adl {} \; the files that are accessed directly are not listed by find. When access is provided to a directory, so that the user can later pick up one or more files, the filesystem seems to be updated, at least partially. Please not that the directive CacheDisable / is being used, and the problem has been reproduced after rebooting the machine and then downloading a single file. At the moment, the computer is running Fedora 8, kernel 2.6.24.5-85.fc8. I am pretty sure I did not notice this problem with old operating system versions like FC2. Problem is reproducible (it always occurs, apparently).
Sounds more like a kernel problem, reassigning. Read ya, Phil
possibly related to the relatime feature, where we only update atime once per day. Eric ?
That'd be my quick guess. Can you boot with default_relatime=0 on the kernel command line and see if the behavior changes? (or alternatively, /proc/sys/kernel/default_relatime can be changed and then the fs in question remounted (maybe umount/mount vs. remount, not sure offhand)).
/proc/sys/kernel/default_relatime does not exist on this system. This is a rather critical server, I'll try to reboot it tomorrow. Please note that your guess does not seem to fit well with what I've seen: the atime in question is not updated even after several days...
> Please note that your guess does not seem to fit well with what I've seen: > the atime in question is not updated even after several days... Hm ok maybe not it then unless it's set to a high interval... sorry, docs were wrong; try: /proc/sys/fs/default_relatime and also maybe see what value is in: /proc/sys/fs/relatime_interval -Eric
Content of /proc/sys/fs/default_relatime is "1" while that of /proc/sys/fs/relatime_interval is "86400" (from the above comments, I'll guess that means one day = 86400 seconds) To set those values permanently equal to zero, is it sufficient to add "default_relatime=0" on grub.conf ? (and reboot...) Paulo
> When access is provided to a single file (for download) the > access time stamp seems not to be updated. Ok, so basically the problem you see is: # stat; chmod; stat doesn't update the atime... Let me double-check this, my understanding is that atime should never be older than ctime or mtime even with relatime. In the meantime, yes, adding the default_relatime=0 parameter to the kernel boot line in grub should turn it off. -Eric
I'm sorry, I said that backwards, trying to do too many things at once today :) With relatime, atime is not updated on access unless it is older than either ctime or mtime. So what exactly is the sequence of operations you're performing on the file which should be updating atime? You just want to see files accessed via http in the past 2 days based on atime? In that case, relatime is behaving as expected and you probably just want to turn it off. Thanks, -Eric
> Ok, so basically the problem you see is: > > # stat; chmod; stat > > doesn't update the atime... Sorry, I don't know the inner workings of either > apache or the kernel... Maybe I should add that if I copy (cp) one of the files accessed through apache to some other place then atime seems to be immediately updated. Paulo PS (after reading your latest post) do you consider OK that atime is not updated by default, or did I read it wrong?
Yes, with relatime, atime is *not* updated on every access, so the behavior you're seeing is expected, I think. Sorry it took me 3 or 4 comments to say that succinctly. :) If mtime or ctime has changed since the last atime update, only *then* is atime refreshed on the next access. But; are you saying that if you "cp /var/www/html/foo /tmp/foo" then the atime on /var/www/html/foo is immediately updated? Now this sounds weird again. -Eric
Yes, I'm saying that if I copy one of the files whose atime is not changed by download through apache, then the "find" command I stated above will immediately pick it up!! Now some more "doubts": Number 1, if relatime is working OK, then why doesn't it update atime even after 1 day? Are the files you told me about simply ignored? (hopefully the kernel boot argument will not be ignored...) Number 2, and this may be a subjective/controversial observation, which I do not wish to be understood as "agressive", rather the result of trying to understand: why is relatime set by default on what could be a typical PC? I am wrong in supposing that only highly loaded servers may really benefit from it? (I am asking this because somehow it would seem more natural to define the default settings in a way that wouldn't change earlier behaviour without good reason) Paulo
> Yes, I'm saying that if I copy one of the files whose atime is not changed > by download through apache, then the "find" command I stated above will > immediately pick it up!! I'm guessing that the copy was > 24 hrs after the last atime update, then. re: #1, on access, the atime will be updated if the current atime is more than 24 hrs old. But, for a single access, it won't magically be updated 24 hrs later. So; cat file; sleep 22 hrs; cat file will not update the atime on the 2nd cat. cat file; sleep 25 hrs; cat file should update the atime on the 2nd cat. #2, well... it just sort of happened in fedora. Atime updates come at fairly large IO expense and can hurt performance, but I tend to agree that it's not the most obvious thing, and not the right answer for everyone. -Eric
I have confirmed the behaviour you report, at least in the case where I copied the same file twice within 1 hour or so apart: atime was not updated the second time. However, I am not sure that can explain everything, even with httpd involved. Could you please leave this bug report open for a couple of weeks, while I give it a closer look now that I know what to expect? Are the (new, not saved) atime values kept in RAM, to be saved when the filesystem is unmounted, eg when the system is shut down? Paulo
The atime updates are completely ignored unless it is deemed time to actually apply the update. If it's not time, they are not saved anywhere. When updated they are updated to the *current* time of the *current* access, not some retroactive update. Sure, I will leave the bug open for a bit. -Eric
Hi Eric, I have done some testing and I still believe there is a problem (one or two). Here's what I've done: Copied a file on 2008-05-20 19:42, and checked that the atime of that file was changed on that access (with ls --time=atime) On 2008-05-21 21:51, hence about 26h afterwards, I downloaded the same file through apache; the atime remained the same as above, 2008-05-20 19:42. One could imagine that somehow (a bug, config mistake, ...) httpd is using a cache, despite the directive "CacheDisable /" that is present in /etc/httpd/conf/httpd.conf So I reboot the system, and downloaded that same file once (still using default values for relatime) --- I did that a couple of hours ago, ie on May 26, approx 18:30 (all times are local). After the download (plus a few "sync" as root) the access time remained at 2008-05-20 19:42 ! So how did apache read the file? OK, there could be a server in between, except that the computer I used to download the file has as nameserver the computer that has the httpd server (and is in a local network controlled by that same server). Moreover: someone accessed a directory today, about nine hours ago, and downloaded two files. Some access times have been changed on that event, and others remain unchanged from 2008-04-04. Now, the atimes that have changed correspond exactly to the files that have NOT been downloaded today through httpd ! (according to httpd/access_log ) This is getting weird... Paulo
In general, without all the added complexity and wondering about apache caches and the like, a basic test seems to confirm that this is working as designed. On an ext3 filesystem: [root@localhost ~]# stat -f . File: "." ID: 368e244c643ff27e Namelen: 255 Type: ext2/ext3 Block size: 4096 Fundamental block size: 4096 Blocks: Total: 8339906 Free: 2013968 Available: 1583479 Inodes: Total: 8617984 Free: 8225262 create a file and look at the atime (stat info filtered below) [root@localhost ~]# touch atime-testfile [root@localhost ~]# stat atime-testfile Access: 2008-05-26 21:08:02.000000000 -0500 and it matches the time when it was created: [root@localhost ~]# date Mon May 26 21:08:15 CDT 2008 now access it and re-stat it, find that atime is unchanged, as expected: [root@localhost ~]# cat atime-testfile [root@localhost ~]# stat atime-testfile Access: 2008-05-26 21:08:18.000000000 -0500 Now set the date forward 24h+ [root@localhost ~]# date 052721152008 Tue May 27 21:15:00 CDT 2008 re-access and re-stat the file, find that it *has* moved forward (old atime was more than 24hrs stale so was updated): [root@localhost ~]# cat atime-testfile [root@localhost ~]# stat atime-testfile Access: 2008-05-27 21:15:05.000000000 -0500 but a subsequent access does not update again: [root@localhost ~]# cat atime-testfile [root@localhost ~]# stat atime-testfile Access: 2008-05-27 21:15:05.000000000 -0500 now advance time again.... [root@localhost ~]# date 052821202008 Wed May 28 21:20:00 CDT 2008 re-access, re-stat, atime is updated: [root@localhost ~]# cat atime-testfile [root@localhost ~]# stat atime-testfile Access: 2008-05-28 21:20:03.000000000 -0500 .... etc. I don't see anything in your scenarios above that absolutely indicate a bug in atime handling. Further: > Moreover: someone accessed a directory today, about nine hours > ago, and downloaded two files. > Some access times have been changed on that event, and others > remain unchanged from 2008-04-04. if one was more than 24hrs stale and the other was not, only the former would be updated. Thanks, -Eric
Just to make sure there was no misunderstanding whatsoever, here is how I read your earlier description of relatime: if, on a given access, the stored atime of a file is older than 24h (default) then that atime is immediately updated to the current time (comment #12). With that in mind, here it goes: > > Moreover: someone accessed a directory today, about nine hours > > ago, and downloaded two files. > > Some access times have been changed on that event, and others > > remain unchanged from 2008-04-04. > > if one was more than 24hrs stale and the other was not, only the > former would be updated. > All the evidence I have is that was *not* the case. Then there is the bit that you did not copy into your post, which says (in other words) that the files downloaded less than 48h ago remained with an access time of over 50 *days* ago, contradicting the expected behaviour described in comment #12. I have found an example where I cannot imagine there can be a cache issue to even consider. A partition of the file server (computer "A") has been exported via nfs to another computer "B". I have just obtained a copy of a file "F" (from A) and stored it at "B" without changing the atime of the original "F", dated from 3 months ago. Here is the relevant output of stat, after the copy has been produced: Access: 2008-02-20 18:44:47.000000000 +0000 Modify: 2007-08-10 00:59:55.000000000 +0100 Change: 2008-02-20 18:48:45.000000000 +0000 How would you explain this? (and I did not mess with the date...) I have already agreed that on the local machine, executing shell commands, things seem OK. But that is not what I have reported. I said file access through httpd seemed to have problems with atime, and now file access through nfs. By the way, this may be a problem beyond relatime (eg, it could be deep in the kernel). If this behaviour cannot be reproduced then I will consider other possibilities, such as this system having been "taken over"... Paulo
Hm, interestingly, even with: echo 0 > /proc/sys/fs/default_relatime on a fedora 9 server, and the same on a fedora 8 client, I still don't see atime updates over nfs with the test you describe. So, that is odd... will look into it. -Eric
if I boot with default_relatime=0 then nfs does properly update. Will dig in a bit later.
There is also the 'norelatime' mount option.
... which apparently doesn't work in the face of default_relatime=1 :(
This message is a reminder that Fedora 8 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 8. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '8'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 8's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 8 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 8 changed to end-of-life (EOL) status on 2009-01-07. Fedora 8 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.