| Summary: | Random "No such file or directory" error on gluster client when using stat-prefetch | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Lee Simpson <lee> | ||||||||
| Component: | stat-prefetch | Assignee: | Raghavendra G <raghavendra> | ||||||||
| Status: | CLOSED WORKSFORME | QA Contact: | |||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | low | ||||||||||
| Version: | 3.0.3 | CC: | amarts, eimamagi, leo, vikas | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | Type: | --- | |||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Attachments: |
|
||||||||||
Created attachment 169 [details]
The main program.
As requested via email, added the following to my client config;
volume trace-below
type debug/trace
subvolumes quickread
end-volume
volume statprefetch
type performance/stat-prefetch
subvolumes trace-below
end-volume
volume trace-above
type debug/trace
subvolumes statprefetch
end-volume
-----
Repeated the test and attached the output of these traces when the problem happens. This time ive also let the trace run a little longer this time, leaving time between refreshes of the awstats page so you can see multiple examples.
Let me know if i can be of any further help.
Created attachment 185 [details] Detailed steps to properly create VNC binaries (as opposed to the ones in the RPM that don't work) Hello, I'm hitting the same/similar problem. I have two CentOS 5.2 servers with 5 partitions each aggregated with cluster/distribute. Gluster version is 3.0.4. I added on client side configuration with trace-above, trace-below from the Comment #1. In my case file cannot be accessed until I go to the directory where problematic file is and perform ls. I attached (gluster-ls-logs.txt) trace log lines which I see after each operation performed. Cheers, emir Regarding Comment #2, the most important thing I forgot to add is that removing: volume statprefetch type performance/stat-prefetch subvolumes quickread end-volume from the client config and performing: umount /data mount /data doesn't solve the problem in my case. The behavior remains the same. Thanks in advance, emir Hi Emir/Lee, Is this bug reproducible in latest releases of 3.0.x (3.0.6) or 3.1.0? regards, Raghavendra. Marking this bug resolved. Please reopen if the issue still persists. |
Gluster client was getting random OPEN/STAT/READ errors on files created by apache (phpbb cache files seemed to cause it most) or awstats.pl run on cron. Apache would be unable to open these files and each time you tried an error would appear in the gluster client logs. SSH'ing into the client and cat'ing the file would resolve the issue and apache would then read it fine. Removing stat-prefetch fixed this issue. Attached is a trace of stat-prefetch and the error happening on the client side. Server logs show nothing when it happens (at normal level). At the moment id rather not put trace onto the server, but if after seeing the client trace you really think its needed, then ill book in some midnight time to do it. It can happen very randomly just with apache doing the writing. However i found running the awstats update script manually caused it to happen ~50% of the time, and proved a good tool to test this problem. STEPS I USED TO REPRODUCE (using 2 gluster clients, a & b.) Client A = awstats script updating program, writting datafiles to gluster. Client B = Webserver using awstats to serve the stats generated by client A 1 - On A, run awstats update script 2 - Quickly point your web browser at B's awstats install AWSTATS then gives ("No such file or directory") error. No matter how long you wait, after you got this error once, you will always get it on that file. Unless you; 3 - SSH into B and Cat the awstats datafile. It displays fine. 4 - Point your web browser at B's awstats install, stats show as expected. Gluster Configs = https://gist.github.com/5f0e6a950712d7f24a64 Example Errors in gluster client logs (at NORMAL log level) = https://gist.github.com/6e2c6dff75fd3bc19fa9 Gluster client/server version = 3.0.3 Kernel = 2.6.31-20-server #58-Ubuntu OS = Ubuntu 9.10 I hope that gives you all the information you need, but im open to doing more tests if required.