Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1965218 - pulp3: satellite-maintain content prepare failed when run as nohup job
Summary: pulp3: satellite-maintain content prepare failed when run as nohup job
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Satellite Maintain
Version: 6.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: 6.9.7
Assignee: Anurag Patel
QA Contact: Gaurav Talreja
URL:
Whiteboard:
Depends On:
Blocks: 1957813
TreeView+ depends on / blocked
 
Reported: 2021-05-27 08:12 UTC by ir. Jan Gerrit Kootstra
Modified: 2021-12-17 21:32 UTC (History)
8 users (show)

Fixed In Version: rubygem-foreman_maintain-0.7.14,tfm-rubygem-katello-3.18.1.45-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-10 16:20:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
the script used to run the required command (34 bytes, text/plain)
2021-05-27 08:12 UTC, ir. Jan Gerrit Kootstra
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 33183 0 Normal Closed pulp3: satellite-maintain content prepare failed when run as nohup job 2021-10-11 14:04:34 UTC
Foreman Issue Tracker 33187 0 Normal Closed pulp3: satellite-maintain content prepare failed when run as nohup job 2021-10-11 14:02:59 UTC
Red Hat Product Errata RHBA-2021:4611 0 None None None 2021-11-10 16:20:53 UTC

Description ir. Jan Gerrit Kootstra 2021-05-27 08:12:41 UTC
Created attachment 1787519 [details]
the script used to run the required command

Description of problem:

nohup ./satellite-maintain-content-prepare.sh > content-prepare.log 2>&1
fails with error:

Failed executing foreman-rake katello:pulp3_migration, exit status 256

content of the script:

satellite-maintain content prepare

Version-Release number of selected component (if applicable):

6.9.2

How reproducible:


Steps to Reproduce:
1. create the script and make it executable
2. run the nohup command
3. tail -f logfile

Actual results:

Failed executing foreman-rake katello:pulp3_migration, exit status 256

Expected results:

no errors as the command does not require input

Additional info:

Comment 1 ir. Jan Gerrit Kootstra 2021-05-27 08:20:30 UTC
running the command in the foreground:

satellite-maintain content prepare

There seems to be no issue.

Comment 2 Peter Vreman 2021-06-17 07:47:38 UTC
A severity Urgent match more the impact on the users starting the migration process.
I also hit this issue directly out-of-the-box when trying the 'satellite-maintian content prepare' step

It is really confusing the users what is happening when using nohup or any other standard bash background process.


I made it work with the following patches
- replace script with tee
- rake task add flush to stdout (otherwise it buffers ~8K)
- add preserve_output to rake command to write multiple lines instead of overwriting the same line

~~~
--- /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/lib/katello/tasks/pulp3_migration.rake.210617-1      2021-06-03 15:05:18.000000000 +0000
+++ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/lib/katello/tasks/pulp3_migration.rake       2021-06-17 07:22:45.320625830 +0000
@@ -24,6 +24,7 @@
         message = "#{Time.now.to_s}: #{task.humanized[:output]}"
         clear_count = message.length + 1
         $stdout.print(message)
+        $stdout.flush

         sleep(10)
         task = ForemanTasks::Task.find(task.id)
--- /usr/share/gems/gems/foreman_maintain-0.7.8/definitions/procedures/content/prepare.rb
+++ /usr/share/gems/gems/foreman_maintain-0.7.8/definitions/procedures/content/prepare.rb
@@ -7,7 +7,7 @@

     def run
       # use interactive to get realtime output
-      puts execute!('foreman-rake katello:pulp3_migration', :interactive => true)
+      puts execute!('foreman-rake katello:pulp3_migration preserve_output=true', :interactive => true)
     end
   end
 end
--- /usr/share/gems/gems/foreman_maintain-0.7.8/foreman_maintain/utils/command_runner.rb
+++ /usr/share/gems/gems/foreman_maintain-0.7.8/lib/foreman_maintain/utils/command_runner.rb
@@ -64,7 +64,7 @@
         # running interactively
         log_file = Tempfile.open('captured-output')
         exit_file = Tempfile.open('captured-exit-code')
-        Kernel.system("script -qc '#{full_command}; echo $? > #{exit_file.path}' #{log_file.path}")
+        Kernel.system("bash -c '#{full_command}; echo $? > #{exit_file.path}' | tee -i #{log_file.path}")
         File.open(log_file.path) { |f| @output = f.read }
         File.open(exit_file.path) do |f|
           exit_status = f.read.strip
~~~

Comment 3 James Jeffers 2021-07-29 16:02:05 UTC
Created redmine issue https://projects.theforeman.org/issues/33183 from this bug

Comment 4 Peter Vreman 2021-08-09 08:55:00 UTC
James,

The problem is that 'script' does not work nice togeher with nohup (or any other standard shell backgrounding from shell) because 'script' changes the 'setsid' and detaches from the parent process when going in the background, then the 'nohup' does not have a sub process to follow anymore

The PR https://github.com/theforeman/foreman_maintain/pull/513 is sadly implemented only half implement the proposed patch. It missing the important part to replace 'script' with standard 'bash' And because 'bash' does not log to files the 'tee' was needed.



Peter

Comment 6 Bryan Kearney 2021-08-25 20:04:50 UTC
Upstream bug assigned to apatel

Comment 7 Bryan Kearney 2021-08-25 20:04:52 UTC
Upstream bug assigned to apatel

Comment 8 Bryan Kearney 2021-09-01 20:05:02 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/33183 has been resolved.

Comment 12 James Jeffers 2021-10-11 14:02:59 UTC
I see where I am confused. The Katello change associated with this was not picked in.

Comment 13 Brad Buckingham 2021-10-11 14:35:57 UTC
Clearing needinfos based upon comment 12

Comment 15 Gaurav Talreja 2021-11-02 13:59:48 UTC
Verified.

Tested on Satellite 6.9.7 Snap 3.0
Version: rubygem-foreman_maintain-0.7.14-1.el7sat.noarch

Steps:
1. echo "satellite-maintain content prepare" > content_prepare.sh
2. nohup ./content_prepare.sh > content-prepare.log 2>&1

Observation:
content prepare command works when run as nohup job without any issues, also works in the foreground.

Change in f-m interactive command runner from `script` to `bash` started causing BZ 2013630, so also tested nohup job with new command runner `stdbuf` and it works fine.

Comment 18 errata-xmlrpc 2021-11-10 16:20:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite Maintenance 6.9.7 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4611


Note You need to log in before you can comment on or make changes to this bug.