We are trying to collect new data from your download files but there aren't new apache logs since more than a month. Until now, we have been collecting your apache logs from a server called download.gluster.org The path where these logs are usually stored is /srv/bitergia/logs/httpd/ I've checked today and these are the latest log files we can see on that path: -rw-r--r-- 1.40G 2017/08/13 10:02:28 download.gluster.org_access.log-20170618 -rw-r--r-- 311.30K 2017/08/21 07:31:58 download.gluster.org_access.log-20170625 -rw-r--r-- 106.58M 2017/08/22 09:30:10 download.gluster.org_access.log-20170702 I remember there was an issue similar to this one on June. In that case, the error was due to the disk was full but I don't know if that's the case this time. This is critical because it is not allowing us to keep data for "Downloads" up to date on your dashboard (https://glusterfs.biterg.io:443/goto/441feba696360f47136f20ad555e7477). Thanks! Regards, David P. -- Copying from email, putting in 'website' because it's ... close?
Seems
Disk is full again. I need to investigate why it grow so much and why the cleanup do not clean the old log. I have added 2G as it should be sufficient for now. The cronjob tonight will make the new data appear.
It also seems that the logs files have grown a lot (there is a lot of 2G file, and some 640m file). I suspect that the server was somehow overloaded and the cleanup jobs didn't cleaned correctly everything for some reasons.
I have improved/fixed the cleanup script ( https://github.com/gluster/gluster.org_ansible_configuration/commit/2355f453429012621855b5995667e214b79898c8 ). Now, it will automatically recompress files that were wrongfully compressed (due to disk being full, OOM errors, etc) and so the problem should fix itself now (instead of seeing the issue persist). I can still imagine some concurrency (like if the server do crash hard at the exact wrong moment and the wrong day), but we are not gonna get a better system without significant time spent on it. I did a manual cleanup a few files, and watched. I may need to later change the partition layout (since we are down at 30G of logs, with 30G of free space for partition ) but since that's xfs, it can't be done online. For what I see, the files are all here, so closing.