Bug 998319 - RPMs put important data in /var/tmp
RPMs put important data in /var/tmp
Status: CLOSED CURRENTRELEASE
Product: JBoss Enterprise Application Platform 6
Classification: JBoss
Component: RPMs (Show other bugs)
6.1.0
Unspecified Unspecified
unspecified Severity urgent
: ER5
: EAP 6.2.0
Assigned To: Permaine Cheung
Katerina Odabasi
Russell Dickenson
:
: 1011067 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-18 23:27 EDT by James Livingston
Modified: 2016-02-21 19:56 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
RPMs put important data in /var/tmp, including /domain/data. /var/tmp is meant to contain temporary data which could be removed at any time. /domain/data should not be redirected to /var/tmp/ since it contains important data that should not be lost, including the transaction recovery journal and HornetQ large message store and journal. This error has been fixed for this release of JBoss EAP 6. For new installation, /usr/share/jbossas/domain/data will be a directory instead of a link. However, for existing installations, one of the following must be performed: * Re-install and migrate the config and data files (including the temp directories) to the new installation directories, or * Uninstall tmpwatch to disable clean up of /var/tmp/jbossas, or * Configure tmpwatch not to remove the files that jbossas puts in to /var/tmp/jbossas
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-15 11:23:15 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description James Livingston 2013-08-18 23:27:51 EDT
The JBoss RPMs have the following symlinks (amongst others):
  /usr/share/jbossas/standalone/tmp -> /var/cache/jbossas/standalone
  /usr/share/jbossas/domain/data -> /var/tmp/jbossas/domain
  /usr/share/jbossas/domain/tmp -> /var/cache/jbossas/domain


domain/data should not be redirected to /var/tmp/ since it contain important data that should not be lost, including the transaction recovery journal and HornetQ large message store and journal.

/var/tmp is meant to contain temporary data which could be removed at any time, although with less frequency than /tmp.

The data symlink should probably be pointed to somewhere under /var/lib/, /var/spool or similar.


If you have tmpwatch installed as well as jbossas-domain, it can result in important but transient JBoss data being removed.
Comment 2 Fernando Nasser 2013-09-06 10:25:45 EDT
/var/spool contains data which is awaiting some kind of later processing.

seems indeed to be the right place for this info.

So

/usr/share/jbossas/domain/data -> /var/spool/jbossas/domain

Do you agree Brian?

The issue now is, if we make this change on an upgrade, the files will be in the former /var/tmp location, and the new location will be empty.  There is no problems with fresh installs.

Does anyone has a suggestion on how to deal with the upgrades?

One possibility would be to use a %trigger on 6.1.1 or older jbossas-domain packages and in it do a 

cp -pr /var/tmp/jbossas/domain /var/spool/jbossas/domain

or something like that.  It is a bit risky though.
Comment 3 Brian Stansberry 2013-09-06 11:07:07 EDT
I'm not an expert on FHS, with most of my knowledge of the rules coming from the brief Wikipedia article.

But from that, /var/lib sounds like a better fit.

"State information. Persistent data modified by programs as they run, e.g., databases, packaging system metadata, etc."

That's what the data/ dirs are used for.

On an intuitive level, /var/spool sounds very wrong. The "contains data which is awaiting some kind of later processing" explanation could be applied to anything that meets the /var/lib definition as well. If it's not relevant for future processing, why store it at all? I think the key here is the word "awaiting". Stuff in /var/spool is expected to be processed further; it is written to make that further processing possible. The stuff in our data/ dirs is written to *allow further processing if needed*. It's a general persistent store, not a set of inputs to a task queue.
Comment 4 Fernando Nasser 2013-09-06 14:38:23 EDT
Taking into consideration Brian's arguments:

/var/lib   holds state information pertaining to an application or the system..

So

/usr/share/jbossas/domain/data -> /var/lib/jbossas/domain

IMPORTANT:

The issue now is, if we make this change on an upgrade, the files will be in the former /var/tmp location, and the new location will be empty.  There is no problems with fresh installs.

Does anyone has a suggestion on how to deal with the upgrades?

One possibility would be to use a %trigger on 6.1.1 or older jbossas-domain packages and in it do a 

cp -pr /var/tmp/jbossas/domain /var/lib/jbossas/domain

or something like that.  It is a bit risky though.
Comment 5 Pavel Janousek 2013-09-09 02:31:13 EDT
What about /var/tmp/* -> /var/tmp/*.oldrpm and newly deployed files to new location (whatever will be decides)? This behavior is visible through 'yum update' and it is up to customer's decision how to solve this state.

I believe, this is easy do-able.
Comment 6 James Livingston 2013-09-09 02:56:42 EDT
I think anything requiring manual intervention would be very dangerous. A lot of people don't read the yum output closely or have automated updates.

Some of the data we are currently placing in the directory is import user data such as the HornetQ paging store. I believe any changes which risk the data going missing (from the user's point of view, even if it's on the disk still) would be a problem. Which is why having in a location where it could be removed like /var/tmp is problematic in the first place.
Comment 7 Fernando Nasser 2013-09-09 09:53:20 EDT
We have two options:

1 - Only use the new location on new installations, not on upgrades

2 - If the new location does not exist, do a

cp -pr /var/tmp/jbossas/domain /var/lib/jbossas/domain

(only if the new location does not exist, so we do not overwrite newer data)


The (1) one does not solve the 'tmp' location for old installations

The (2) is a bit dangerous too -- moving important user files around is frill with peril


Another option would be to ask users to make a change to /etc/cron.daily/tmpwatch to skip cleaning /var/tmp/jbossas in a release notes.  This combined with (1) perhaps.  


P.S.: Some files in /var/tmp/jbossas are only lost if the server stays shut down for more than 30 days.  For HornetQ service is not started for more than 30 days or if it does not change the large message one (well, this msg not delivered for 30 days is also not normal).  I guess that is why this did not came up earlier.
Comment 8 Fernando Nasser 2013-09-24 14:14:16 EDT
Hi James, what do you think about my last comment?  

We can also release note and ask customers to disable the cleanup of /var/tmp/jbossas for old installations, this way we can go with option 1 and avoid meddling with existing files.

Any other suggestions?
Comment 9 James Livingston 2013-09-24 22:09:28 EDT
For (1), that means we simply change where the symlink points to in the package, but don't change an already existing one?  That sounds fine to me.
Comment 10 Fernando Nasser 2013-09-25 09:30:18 EDT
That is correct.  But for an existing customer, including the one were the issue occurred, we'l need to ask them to do one of two things (release notes topic):

1) Do a fresh install and migrate their config and data to there, including the temp directories the AS creates (some customers have script for this, others don't).

2) Stop cleaning of /var/tmp/jbossas (if cleaning of /var/tmp enabled at all)

So, IYO should we proceed with (1)?

Repeating here for clarity:

1 - Only use the new location on new installations, not on upgrades
Comment 11 James Livingston 2013-09-27 00:41:55 EDT
That all sounds fine to me.
Comment 13 David Walluck 2013-10-01 18:09:16 EDT
I noticed that someone else opened a similar bug report a little bit after this one and you may want to mark it as a duplicate (I put it in `See Also' for now).

Note that the report suggests that this issue is due to a simple mistake whereby the data/ and tmp/ directory symlinks were swapped. If you switch them around, you will then have the data/ pointing to /var/cache/ and the tmp/ pointing to /var/tmp/ and then the locations actually make a lot more sense. I did not know which prior release to check in order to see if the reverse had actually been the case sometime in the past.

Please do not overlook the fact that tmp/ may be pointing to the wrong place now as well and consider /var/cache as a location for data/ if it is the case that the data/ may be deleted without data loss.
Comment 14 dereed 2013-10-01 18:39:28 EDT
> and consider /var/cache as a location for data/ if it is the case that the 
> data/ may be deleted without data loss.

That is not the case.
Comment 15 David Walluck 2013-10-01 18:52:46 EDT
OK, and(In reply to dereed from comment #14)
> > and consider /var/cache as a location for data/ if it is the case that the 
> > data/ may be deleted without data loss.
> 
> That is not the case.

Thanks. And is there a case for leaving tmp/ in /var/cache instead of /var/tmp?

/var/tmp : Temporary files preserved between system reboots
/var/cache : Application cache data
Comment 16 Fernando Nasser 2013-10-02 10:02:01 EDT
David, please see comment #3 where Brian explains what type of data it is.

/var/tmp can go away, which is what hurt in the case that generated the issue

/var/cache requires that "The application must be able to regenerate or restore the data."  which is not the case (if it were, we would not have had the problem when the files in /var/tmp were lost).  BTW, the data that fits into this case already goes into /var/cache/jbossas/domain 

So it is /var/lib as Brian said.
Comment 17 David Walluck 2013-10-02 13:20:49 EDT
(In reply to Fernando Nasser from comment #16)
> David, please see comment #3 where Brian explains what type of data it is.

The discussion has been about data/ which is currently pointing to /var/tmp/. I agree that the best place for that sounds like /var/lib/.

I was asking about tmp/ which is currently pointing to /var/cache/, not /var/tmp/, even though they both have tmp/ in their name. I do not think that any comment has mentioned this directory yet.
Comment 18 Fernando Nasser 2013-10-02 14:04:47 EDT
the tmp link is correct. For that, "The application must be able to regenerate or restore the data." of /var/cache holds true.
Comment 19 Fernando Nasser 2013-10-06 12:31:45 EDT
*** Bug 1011067 has been marked as a duplicate of this bug. ***
Comment 20 Permaine Cheung 2013-10-10 11:24:49 EDT
The fix has been commited, and will be in ER5.
For new installation, /usr/share/jbossas/domain/data will be a directory instead of a link (because /usr/share/jbossas/domain/ is a link to /var/lib/jbossas/domain which is where we want to link to, hence instead of linking, we create the directory there). Nothing changes on an upgrade.
Comment 21 Pavel Janousek 2013-10-15 08:19:39 EDT
Fixed in ER5 build as described by Permaine in comment #20.
Comment 22 Fernando Nasser 2013-10-15 12:09:33 EDT
Hi James, should we have some release notes text for this?  Some instructions on how to migrate the data from the old location perhaps?
Comment 23 James Livingston 2013-10-15 22:33:57 EDT
That sounds like a good idea. Especially since we aren't migrating it on upgrade, telling people how to do it manually sounds good.
Comment 26 James Livingston 2013-12-02 22:05:23 EST
You can 1) uninstall tmpwatch, or 2) configure it not to remove the files JBoss puts there

Note You need to log in before you can comment on or make changes to this bug.