Bug 925983

Summary: Systemd private tmp files break Wordpress and mysqld
Product: [Fedora] Fedora Reporter: Greg Scott <gregscott>
Component: systemdAssignee: systemd-maint
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: extras-orphan, johannbg, lnykryn, metherid, mschmidt, msekleta, notting, plautrba, systemd-maint, tgl, vpavlin, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-25 07:22:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Greg Scott 2013-03-23 03:07:07 UTC
Description of problem:
See Bugzilla number 782513 at https://bugzilla.redhat.com/show_bug.cgi?id=782513

I think the action taken for that Bugzilla broke mysqld.

When mysqld uses a private tmp directory assigned by systemd, it runs just fine for a few days.  Apparently, systemd has a facility to clean or remove private /tmp directories it knows about.  When it removes the private tmp directory for mysqld, mysqld can no longer write to its tmp directory.  This, in turn, breaks Wordpress, which depends on mysqld for the database describing web pages.  It may also break other products that depend on mysql.

This is a particularly nasty bug because everything works for, apparently, 10 days before the problems start to occur.  The only footprints the problem leaves behind are entries in /etc/httpd/logs/error_log that look like this:

[Fri Mar 22 17:01:44.427009 2013] [:error] [pid 20925] [client 69.171.247.116:40927] WordPress database error Can't create/write to file '/tmp/#sql_435_0.MYI' (Errcode: 2) for query SELECT t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('category', 'post_tag', 'post_format') AND tr.object_id IN (267) ORDER BY t.name ASC made by require('wp-blog-header.php'), wp, WP->main, WP->query_posts, WP_Query->query, WP_Query->get_posts, _prime_post_caches, update_post_caches, update_object_term_cache, wp_get_object_terms

Meanwhile, Wordpress pages and blog posts start to behave unpredictably because the database they depend on is crippled.  See this URL for more details:

http://stackoverflow.com/questions/11997012/mysql-cant-create-write-to-file-tmp-sql-3c6-0-myi-errcode-2-what-does


Version-Release number of selected component (if applicable):

mysqld and systemd that ship with Fedora 18

How reproducible:

At will

Steps to Reproduce:
1.  Set up a Wordpress website on a F18 system.
2.  Let it run for several days.
3.  After 10 days, type tail /etc/httpd/logs/error_log -f
4.  Access pages from the website and watch the errors fly.
4.  Restart mysqld and repeat steps 2-4.  

Actual results:

Database errors any time anyone anywhere touches any page of the hosted website, but only after things run smoothly for several days.  After the tmp directory goes away, unpredictable behavior with various Wordpress functions that just suddenly break for no apparent reason.  Significant loss of sleep for website developers and Wordpress support staff.

Expected results:

The tmp directory mysqld depends on should not disappear underneath mysqld; if that tmp directory stays intact, mysqld will not break and products that depend on mysqld will not break.  

Additional info:

I am indebted and grateful for the answers provided in
http://stackoverflow.com/questions/11997012/mysql-cant-create-write-to-file-tmp-sql-3c6-0-myi-errcode-2-what-does

When I restarted mysqld on my Fedora 18 web host, the database errors stopped and my brand new website blog entries started displaying again.

Comment 1 Tom Lane 2013-03-23 04:43:48 UTC
Since no such complaints were seen before F18, and mysql hasn't changed behavior, I'm thinking something in systemd broke.

Comment 2 Michal Schmidt 2013-03-25 07:22:55 UTC
This has been fixed upstream for bug 866693. The fix is in v198. There will be a rebase or a backport to F18 soon.

*** This bug has been marked as a duplicate of bug 866693 ***