Bug 2217397
| Summary: | REX job finished with exit code 0 but the script failed on client side due to no space. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Hao Chang Yu <hyu> | ||||||||
| Component: | Remote Execution | Assignee: | Adam Ruzicka <aruzicka> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Peter Ondrejka <pondrejk> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 6.11.5 | CC: | ahumbe, aruzicka, balu.shanmugam, ccordoui, iballou, rlavi, vdeshpan | ||||||||
| Target Milestone: | 6.15.0 | Keywords: | Triaged | ||||||||
| Target Release: | Unused | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | rubygem-smart_proxy_remote_execution_ssh-0.10.2 | Doc Type: | If docs needed, set a value | ||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | |||||||||||
| : | 2246551 2250342 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2024-04-23 17:11:32 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Redhat confirms it is a bug fixed on version 6.136. We are in version 6.11. We need a hot fix for this version as well as 6.12. This bug is heavily impacting our patching, and we cannot wait for a migration to solve this. Hello Balu, Thank you for the information, I tried understanding it but could not see any updates from our Engineering team about the fix being available or bug being resolved in Satellite 6.13, can you help us with the source or steps you have received so we can test too? Regards, Vedashree Deshpande. This is definitely not fixed on 6.13. Created redmine issue https://projects.theforeman.org/issues/36655 from this bug @Vedashree Deshpande, I am not sure what source and steps you are talking about? Please refer #Case 03542621 for more details. Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36655 has been resolved. (In reply to balu.shanmugam from comment #5) > @Vedashree Deshpande, I am not sure what source and steps you are talking > about? > Please refer #Case 03542621 for more details. Sure, I got the answer of how you reproduced the issue. Thank you. Created attachment 1994864 [details]
RHEL 7 Hotfix RPM for Satellite 6.11.5
A Hotfix RPM is now available for Satellite 6.11.5 on RHEL 7.
Installation instructions:
1. Take a backup or snapshot of the Satellite server.
2. Download the RHEL 7 hotfix RPM from the attachment.
3. # yum localinstall ./tfm-rubygem-smart_proxy_remote_execution_ssh-0.5.3-2.HOTFIXRHBZ2217397.el7sat.noarch.rpm --disableplugin=foreman-protector
4. # satellite-maintain service restart
Created attachment 1994865 [details]
RHEL 8 Hotfix RPM for Satellite 6.11.5
Installation instructions (RHEL 8):
1. Take a backup or snapshot of the Satellite server.
2. Download the RHEL 8 hotfix RPM from the attachment.
3. # dnf install ./rubygem-smart_proxy_remote_execution_ssh-0.5.3-2.HOTFIXRHBZ2217397.el8sat.noarch.rpm --disableplugin=foreman-protector
4. # satellite-maintain service restart
Created attachment 1994866 [details]
RHEL 8 Hotfix RPM for Satellite 6.12.5
A Hotfix RPM is now available for Satellite 6.12.5 on RHEL 8.
Installation instructions:
1. Take a backup or snapshot of the Satellite server.
2. Download the hotfix RPM from the attachment
3. # dnf install ./rubygem-smart_proxy_remote_execution_ssh-0.7.3-2.HOTFIXRHBZ2217397.el8sat.noarch.rpm
4. # satellite-maintain service restart
Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set. Verified in stream snap 36 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:2010 |
Description of problem: Satellite shows the REX job has finished successfully but the script was actually failed with no space available on the client. Based on the Dynflow task output below, the wrapper script failed to write the exit code to a file due to no space which caused the terminal to exit with 0. ---------------------- proxy_output: result: <snip> - output_type: stdout <snip> Error Summary ------------- Disk Requirements: At least XXXX more space needed on the / filesystem. <=========================== Yum failed due to insufficient space Uploading Enabled Repositories Report Loaded plugins: product-id, subscription-manager timestamp: xxxxxxx - output_type: stdout output: | Package action failed, exiting... sh: line 0: echo: write error: No space left on device <============================ Wrapper script failed to write the exit code to the file timestamp: xxxxxxx runner_id: xxxxxx exit_status: 0 <==================== wrong exit code. ---------------------- Additional info: Based on "sh: line 0: echo: write error: No space left on device" error above, I think the shell script which wrapped the command failed to redirect the Yum exit code to the "@exit_code_path" file due to completely ran out of space in "/" directory. Since the @exit_code_path" file is empty, the terminal exited with 0 status code. -------------------------------- <<-SCRIPT.gsub(/^\s+\| /, '') | sh -c "(#{@user_method.cli_command_prefix}#{su_method ? "'#{@remote_script} < /dev/null '" : "#{@remote_script} < /dev/null"}; echo \\$?>#{@exit_code_path}) | /usr/bin/tee #{@output_path} <====================== redirect the YUM exit code to a file | exit \\$(cat #{@exit_code_path})" <=============== exit the script with the exit code in the file SCRIPT -------------------------------- I think we should be able to prevent this issue by checking the exit status of the wrapping script itself in the case that the wrapping script itself fail to write the exit code of the Yum command.