Description of problem: If the directory or file system where the apply-updates utility is extracted has insufficient space or ACL rights, the apply-updates command can leave the agent or server in a corrupted state. This is due to the backup failing. For example, the move command is not able to actually write the file in its destination so it therefore doesn't remove it from its source. This means that the files in the file removal list do not actually get removed when the update is applied. Version-Release number of selected component (if applicable): 3.2.3 How reproducible: Always Steps to Reproduce: 1. Install JBoss ON 3.2. 2. Extract the update-03 utility to a location that has less then 300 MB of free space. 3. Run the apply-updates command on the JBoss ON installation. Actual results: Update did not get properly applied and old copies of API or other important lib files were left in place allowing them to be picked up post-update. update.log is incomplete and/or contains messages similar to: mv /opt/jboss/jboss-on/jon-server/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-core-domain-ejb3.jar/org/rhq/core/domain/measurement/MeasurementOOB.java /data/jon-server-3.2.0.GA-update-03/.old/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-core-domain-ejb3.jar/org/rhq/core/domain/measurement mv /opt/jboss/jboss-on/jon-server/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-agent/rhq-enterprise-agent-4.9.0.JON320GA.jar /data/jon-server-3.2.0.GA-update-03/.old/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-agent mv: error writing ‘/data/jon-server-3.2.0.GA-update-03/.old/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-agent/rhq-enterprise-agent-4.9.0.JON320GA.jar’: No space left on device mv: failed to extend ‘/data/jon-server-3.2.0.GA-update-03/.old/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-agent/rhq-enterprise-agent-4.9.0.JON320GA.jar’: No space left on device mv /opt/jboss/jboss-on/jon-server/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-client/rhq-remoting-cli-4.9.0.JON320GA.zip /data/jon-server-3.2.0.GA-update-03/.old/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-client mv: error writing ‘/data/jon-server-3.2.0.GA-update-03/.old/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-client’: No space left on device mv: failed to extend ‘/data/jon-server-3.2.0.GA-update-03/.old/modules/org/rhq/server-startup/ma Expected results: No failure messages and file system in agent and server match the expected contents for a post-updated component. Additional info: First, the backup should not be written to the utilities location. Instead, it should be stored in the target components base directory either in the .patched location with a date or other indication of when it was executed--perhaps even the patch it is associated with. Second, if the backup fails, perhaps the upgrade should fail. This is where a --force option comes in handy or other command-line argument that can tell the update to continue without backing up. Third, instead of the mv, a cp should be done followed by a delete. We should also check to see if the file exists before attempting the copy.
Moving into CR01 target milestone as missed ER01 cutoff.
Moving to ON_QA as available to test with latest CP build: http://download.devel.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/16/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip *Note: jon-server-patch-3.3.0.GA.zip maps to CR01 build of jon-server-3.3.0.GA-update-01.zip.
unable to upgrade and/or start JON after getting "no space left" while applying 3.3.1 patch on top of JON 3.2.0 GA log attached
Created attachment 992729 [details] apply_patch_no_space_left_3.2_3.3.1
Update should have failed at first sign of failure. However, I want to be clear that the expectation is: * apply updates... * failed due to insufficient space... * apply updates is aborted due to error... * clean up space so there is plenty of free space... * apply updates... * all is well and server/agent can now be started. This is compared to what I see in the attached log of: * apply updates... * failed due to insufficient space... * clean up space so there is plenty of free space... * all is well and server/agent can now be started.
Created attachment 992798 [details] apply_patch_no_space_left_3.3.0_3.3.1
This is fixed with the following commits(in order): 6a1c07a59e241bf33 9e493616fd5646c49 f458292a927cf565b The fix involved better checking of exit/return codes during file modification operations and early exit when possible. Additionally tightened up the JON server version checking. Moving to MODIFIED and CR02 for testing in the next build.
Moving to ON_QA as available to test with latest CP build: http://download.devel.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/19/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip *Note: jon-server-patch-3.3.0.GA.zip maps to CR02 build of jon-server-3.3.0.GA-update-01.zip.
Created attachment 993611 [details] 3.2_3.3.1_apply_patch.log
Created attachment 993613 [details] 3.3.0_3.3.1_apply_patch_no_permission
Created attachment 993614 [details] 3.3.0_3.3.1_apply_patch_no_space_left_correct.log
1. apply patch 3.3.1 to 3.2 is not possible - correct - log attached 2. apply patch 3.3.1 to 3.3.0 with no free space does not break existing 3.3.0 - correct - log attached 3. apply patch 3.3.1 to 3.3.0 without permissions - each and every try creates log without while .patched/3.3.0.GA-update-$date/update.log file is not actually created - lot attached
Created attachment 993719 [details] 3.3.0_3.3.1_apply_patch_win_success
Created attachment 993888 [details] 3.3.0_3.3.1_apply_patch_win_no_permission