Bug 1486862

Summary:	Apache Logs are missing and we can't get the dashboard to work from Bitergia.
Product:	[Community] GlusterFS	Reporter:	Amye Scavarda <amye>
Component:	project-infrastructure	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, gluster-infra, mscherer
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-30 20:04:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Amye Scavarda 2017-08-30 16:04:42 UTC

We are trying to collect new data from your download files but there aren't new apache logs since
more than a month.

Until now, we have been collecting your apache logs from a server called download.gluster.org
The path where these logs are usually stored is /srv/bitergia/logs/httpd/

I've checked today and these are the latest log files we can see on that path:
-rw-r--r--            1.40G   2017/08/13   10:02:28   download.gluster.org_access.log-20170618
-rw-r--r--        311.30K    2017/08/21   07:31:58   download.gluster.org_access.log-20170625
-rw-r--r--        106.58M   2017/08/22   09:30:10   download.gluster.org_access.log-20170702

I remember there was an issue similar to this one on June. In that case, the error was due to the disk
was full but I don't know if that's the case this time.

This is critical because it is not allowing us to keep data for "Downloads" up to date on your dashboard
(https://glusterfs.biterg.io:443/goto/441feba696360f47136f20ad555e7477).

Thanks!

Regards,
David P.


-- 
Copying from email, putting in 'website' because it's ... close?

Comment 1 M. Scherer 2017-08-30 16:58:13 UTC

Seems

Comment 2 M. Scherer 2017-08-30 17:01:20 UTC

Disk is full again. I need to investigate why it grow so much and why the cleanup do not clean the old log. I have added 2G as it should be sufficient for now. The cronjob tonight will make the new data appear.

Comment 3 M. Scherer 2017-08-30 17:50:32 UTC

It also seems that the logs files have grown a lot (there is a lot of 2G file, and some 640m file). I suspect that the server was somehow overloaded and the cleanup jobs didn't cleaned correctly everything for some reasons.

Comment 4 M. Scherer 2017-08-30 20:03:47 UTC

I have improved/fixed the cleanup script ( https://github.com/gluster/gluster.org_ansible_configuration/commit/2355f453429012621855b5995667e214b79898c8 ). 

Now, it will automatically recompress files that were wrongfully compressed (due to disk being full, OOM errors, etc) and so the problem should fix itself now (instead of seeing the issue persist). I can still imagine some concurrency (like if the server do crash hard at the exact wrong moment and the wrong day), but we are not gonna get a better system without significant time spent on it. 

I did a manual cleanup a few files, and watched.

I may need to later change the partition layout (since we are down at 30G of logs, with 30G of free space for partition ) but since that's xfs, it can't be done online.

For what I see, the files are all here, so closing.