Bug 1329068

Summary:	MySQL cannot write to /tmp directory
Product:	OpenShift Online	Reporter:	(. )( .) <d.wisskirchen>
Component:	Image	Assignee:	Sally <somalley>
Status:	CLOSED NOTABUG	QA Contact:	Wenjing Zheng <wzheng>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.x	CC:	aos-bugs, d.wisskirchen, jokerman, mmccomas, rthrashe
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-05-25 06:55:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description (. )( .) 2016-04-21 06:03:12 UTC

Description of problem:

When querying MySQL for a larger dataset that does not seem to fit into memory it attempts to store the result into a temporary file. While doing so, I am getting the following error:

SQL Error: 3, SQLState: HY000
Error writing file '/tmp/MYQAa9R1' (Errcode: 122)
could not extract ResultSet; nested exception is org.hibernate.exception.GenericJDBCException: could not extract ResultSet


When I ssh into the machine I can create files in /tmp up to 560MB, so I think there is enough space on the machine.

Chmod'ing the /tmp to 777 does not help.

Also, I cannot reconfigure the tmpdir in mysql/conf/my.cnf because this file is regenerated from the my.cnf.erb.hidden template at every deployment.


Version-Release number of selected component (if applicable): 
current production version


How reproducible:

Simply open https://padel.koeln/ranking/teams. This will cause the application to generate a sql query that does not fit into memory and a generic catchall error page will be shown

Currently running on ex-std-node593.prod.rhcloud.com


Actual results:
query does not fit into memory and cannot be written to disk

Expected results:
query should be written to disk if too large to fit into memory

Comment 1 Sally 2016-05-24 15:17:37 UTC

It looks like this app has changed since you made this report, the link above now gives a 'not found' error.  Can you confirm if this is still a problem?

Comment 2 (. )( .) 2016-05-24 15:23:35 UTC

I can confirm that the app has changed and that the bug still exists. 

The easiest way to reproduce this is to generate a SQL query that is too large to be handled in memory.

I worked around the issue by splitting one query into multiple queries, but the underlying bug still exists.

Comment 3 Sally 2016-05-24 21:56:04 UTC

I was able to reproduce this by setting up an app with MySQL that approaches the gear disk quota (1GB).  I have no problem writing query results to '/tmp' until the quota is reached.  Seems the problem is your app is approaching it's disk usage limit and the query pushes it to it's max when it writes to '/tmp' dir, and therefore can't be completed.  

What's the output of:
$ rhc ssh <app>
# quota

Thanks.

Comment 4 (. )( .) 2016-05-25 06:55:26 UTC

rhc app-show --gear
Gear                     Cartridges               Used Limit
------------------------ ---------------------- ------ -----
54da6eaa4382ec9b55000029 jbossews-2.0 mysql-5.5 0.6 GB  1 GB


$ rhc ssh <app>
# quota
Disk quotas for user 54da6eaa4382ec9b55000029 (uid 3328):
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/EBSStore01-user_home01
                 562336       0 1048576           13065       0   80000

Even though this looks like there is more than enough free space for a mysql query to be stored to /tmp I think you could be right, because the original query was outer joining mulitple tables.

I am going to close this issue. Sorry for the trouble and thank you for your support!

Comment 5 Sally 2016-05-25 13:55:39 UTC

Ok, np :)  <--preferred ascii art, btw ;)