Bug 2217397 - REX job finished with exit code 0 but the script failed on client side due to no space. [NEEDINFO]
Summary: REX job finished with exit code 0 but the script failed on client side due to...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Remote Execution
Version: 6.11.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: Unspecified
Assignee: Adam Ruzicka
QA Contact: Satellite QE Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-06-26 07:33 UTC by Hao Chang Yu
Modified: 2023-08-09 04:46 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
balu.shanmugam: needinfo? (balu.shanmugam)
balu.shanmugam: needinfo? (vdeshpan)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 36655 0 Normal New REX job finished with exit code 0 but the script failed on client side due to no space. 2023-08-08 15:06:50 UTC
Red Hat Issue Tracker SAT-19435 0 None None None 2023-08-08 15:10:47 UTC

Description Hao Chang Yu 2023-06-26 07:33:29 UTC
Description of problem:
Satellite shows the REX job has finished successfully but the script was actually failed with no space available on the client.

Based on the Dynflow task output below, the wrapper script failed to write the exit code to a file due to no space which caused the terminal to exit with 0.
----------------------
proxy_output:
  result:
    <snip>
  - output_type: stdout
      <snip>
      Error Summary
      -------------
      Disk Requirements:
        At least XXXX more space needed on the / filesystem. <=========================== Yum failed due to insufficient space

      Uploading Enabled Repositories Report
      Loaded plugins: product-id, subscription-manager
    timestamp: xxxxxxx
  - output_type: stdout
    output: |
      Package action failed, exiting...
      sh: line 0: echo: write error: No space left on device  <============================ Wrapper script failed to write the exit code to the file
    timestamp: xxxxxxx
  runner_id: xxxxxx
  exit_status: 0   <==================== wrong exit code.
----------------------




Additional info:

Based on "sh: line 0: echo: write error: No space left on device" error above, I think the shell script which wrapped the command failed to redirect the Yum exit code to the "@exit_code_path" file due to completely ran out of space in "/" directory. Since the @exit_code_path" file is empty, the terminal exited with 0 status code.

--------------------------------
      <<-SCRIPT.gsub(/^\s+\| /, '')
      | sh -c "(#{@user_method.cli_command_prefix}#{su_method ? "'#{@remote_script} < /dev/null '" : "#{@remote_script} < /dev/null"}; echo \\$?>#{@exit_code_path}) | /usr/bin/tee #{@output_path}   <====================== redirect the YUM exit code to a file
      | exit \\$(cat #{@exit_code_path})"   <=============== exit the script with the exit code in the file
      SCRIPT
--------------------------------

I think we should be able to prevent this issue by checking the exit status of the wrapping script itself in the case that the wrapping script itself fail to write the exit code of the Yum command.

Comment 1 balu.shanmugam 2023-07-20 05:14:35 UTC
Redhat confirms it is a bug fixed on version 6.136.
We are in version 6.11. We need a hot fix for this version as well as 6.12.
This bug is heavily impacting our patching, and we cannot wait for a migration to solve this.

Comment 2 Vedashree Deshpande 2023-08-08 06:33:42 UTC
Hello Balu, 

Thank you for the information, I tried understanding it but could not see any updates from our Engineering team about the fix being available or bug being resolved in Satellite 6.13, can you help us with the source or steps you have received so we can test too?

Regards, 
Vedashree Deshpande.

Comment 3 Adam Ruzicka 2023-08-08 15:05:50 UTC
This is definitely not fixed on 6.13.

Comment 4 Adam Ruzicka 2023-08-08 15:06:49 UTC
Created redmine issue https://projects.theforeman.org/issues/36655 from this bug

Comment 5 balu.shanmugam 2023-08-09 04:46:34 UTC
@Vedashree Deshpande, I am not sure what source and steps you are talking about?
Please refer #Case 03542621 for more details.


Note You need to log in before you can comment on or make changes to this bug.