Bug 752550
| Summary: | agent not cleaning up old downloaded bundle version files | ||
|---|---|---|---|
| Product: | [Other] RHQ Project | Reporter: | wertnick |
| Component: | Agent | Assignee: | John Mazzitelli <mazz> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Mike Foley <mfoley> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.0.0 | CC: | hrupp, mazz, skondkar |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-02-07 19:29:46 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 760116 | ||
I think I've found the real root cause of the problem. Your agents store bundle deployments in data/tmp, but data/tmp is never cleared, leading to the mount that this is stored on to fill up. Please advise. I guess there are two options I can see. Maybe you have a third. 1: periodically restart the agent with a --purgedata. 2: Some sort of JON call from the server that will purge the data (I don't see this command anywhere) 3: Something else? yes, we'll have to implement some type of cleanup. the bundle files in data/tmp (IIRC) are not used once the bundle is fully deployed, so I think they can be purged. master commit: 16823b9934ea91e81e0b7102bd464a57875fe033 the agent will now purge all old downloaded files from the tmp directory to test: 1) start an agent 2) upload a bundle to the server, and prepare to deploy the bundle to the agent machine 3) before you deploy the bundle, create a temporary directory/file in the agent's data/tmp/bundle-versions directory: $ mkdir -p $RHQ_AGENT_HOME/data/tmp/bundle-versions/blah $ echo abc > $RHQ_AGENT_HOME/data/tmp/bundle-versions/blah/hello.txt 4) deploy the bundle after the bundle is done deploying, you should see the "blah" directory you just created get deleted. You will see a new directory created under bundle-versions (with a name like "10001"). This is fine - this is the latest bundle that was downloaded. We leave this be mainly for debugging purposes (in case something goes wrong, you can go in here and see what files actually got pulled down). You should only ever have one directory under bundle-versions, no matter how many bundles you've deployed in the past. Verified on master build#823 (Version: 4.3.0-SNAPSHOT Build Number: d7ef96e) Followed the steps and verified that the "blah" directory in $RHQ_AGENT_HOME/data/tmp/bundle-versions get deleted after bundle deployment and a new directory with a name "10001" with deployed files is created under 'bundle-versions' directory. Deployed bundles multiple times and verified that 'bundle-versions' directory has only one directory with latest bundle files and old directories with deployed files get deleted. Marking as verified. changing status of VERIFIED BZs for JON 2.4.2 and JON 3.0 to CLOSED/CURRENTRELEASE marking VERIFIED BZs to CLOSED/CURRENTRELEASE |
Description of problem: We have a job that uses JON to deploy every 20 minutes. Over a day or so, the bundle will eventually fail its deployment with the following error. I replaced the changing filename with <somefile>. The error goes away after re-setting the agent configuration (rhq-agent.sh -l). java.io.FileNotFoundException: data/tmp/bundle-versions/12322/<somefile> (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:179) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.rhq.core.pc.bundle.BundleManager.downloadBundleFiles(BundleManager.java:267) at org.rhq.core.pc.bundle.BundleManager.access$000(BundleManager.java:75) at org.rhq.core.pc.bundle.BundleManager$1.run(BundleManager.java:162) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Version-Release number of selected component (if applicable): How reproducible: This seems to pretty consistently occur, but it takes quite a few deploys before this happens. Steps to Reproduce: 1. Start deploying a bundle to an instance repeatedly for a day or two. 2. Wait for a failed deploy. 3. Check the reason: FileNotFoundException. Actual results: A failed deploy. Expected results: A working deploy, like the other 50 or so times I deployed. Additional info: None.