Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1967685 - After upgrade, some of the satellite services failed to start
Summary: After upgrade, some of the satellite services failed to start
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Satellite Maintain
Version: 6.10.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: 6.10.0
Assignee: Justin Sherrill
QA Contact: Devendra Singh
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-03 15:22 UTC by Devendra Singh
Modified: 2023-09-15 01:08 UTC (History)
9 users (show)

Fixed In Version: rubygem-foreman_maintain-0.8.3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 13:48:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 32870 0 Normal Closed satellite 6.10 upgrade should refresh detected features before restarting services after upgrade 2021-07-06 09:57:39 UTC
Red Hat Product Errata RHBA-2021:4697 0 None None None 2021-11-16 13:48:12 UTC

Description Devendra Singh 2021-06-03 15:22:51 UTC
Description of problem: After upgrade, some of the satellite services failed to start


Version-Release number of selected component (if applicable):
6.10 Snap3

How reproducible:
always

Steps to Reproduce:
1. Install the 6.9.z satellite
2. Run the pulp migration and it completes successfully.
3. Run the upgrade from 6.9.z to 6.10 Snap3.
4. Upgrade completes successfully.
5- In the post-upgrade task, Some of the services unable to restart.

Starting the following service(s):
pulp_celerybeat, pulp_resource_manager
\ All services started                                                          
| Server responded successfully!                                      [OK]      
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check whether all services are running:                               [FAIL]
Following services are not running: pulp_celerybeat, pulp_resource_manager, smart_proxy_dynflow_core


Actual results:
Some of the services failed to start in the post-upgrade task

Expected results:
All the services should start successfully.

Additional info:

Comment 3 wclark 2021-06-08 15:51:43 UTC
`Following services are not running: pulp_celerybeat, pulp_resource_manager, smart_proxy_dynflow_core`

These services are removed in Satellite 6.10, so the fix should be to remove them from foreman-maintain

Comment 9 Justin Sherrill 2021-06-17 19:52:36 UTC
Are you sure that the satellite actually upgraded to 6.10?  I tried this and ran into a ton of dependency errors (https://bugzilla.redhat.com/show_bug.cgi?id=1972995).  Looking at your output, i don't see packages getting updated or the installer run, was your output just part of it?

Comment 10 Devendra Singh 2021-06-18 06:00:56 UTC
(In reply to Justin Sherrill from comment #9)
> Are you sure that the satellite actually upgraded to 6.10?  I tried this and
> ran into a ton of dependency errors
> (https://bugzilla.redhat.com/show_bug.cgi?id=1972995).  Looking at your
> output, i don't see packages getting updated or the installer run, was your
> output just part of it?

This problem was happened on 6.10 Snap3, on 6.10 snap5 upgrade gets failed due to a package dependency issue.

Comment 11 Justin Sherrill 2021-06-23 18:31:02 UTC
Devendra, did you see amit's question:  


Devendra can you please confirm if you run : foreman-maintain service restart explicitly after the upgrade do you see pulp2 services getting restarted? and its specifically happening in the upgrade run's post actions?

Comment 12 Justin Sherrill 2021-06-23 21:53:05 UTC
I kind of reproduced this and kind of didnt' reproduce this:

SNIP
Upgrade Step 3/5: katello:upgrades:4.0:remove_ostree_puppet_content. =============================================
Upgrade Step 4/5: katello:upgrades:4.1:sync_noarch_content. =============================================
Upgrade Step 5/5: katello:upgrades:4.1:fix_invalid_pools. I, [2021-06-23T21:46:57.541313 #21212]  INFO -- : Corrected 0 invalid pools
I, [2021-06-23T21:46:57.541350 #21212]  INFO -- : Removed 0 orphaned pools
  Success!
  * Satellite is running at https://610upgradetest.windhelm.example.com

  * To install an additional Capsule on separate machine continue by running:

      capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE" --certs-tar "/root/$CAPSULE-certs.tar"
  * Capsule is running at https://610upgradetest.windhelm.example.com:9090

  The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.
                                      [OK]
--------------------------------------------------------------------------------
Execute upgrade:run rake task:                                        [OK]
--------------------------------------------------------------------------------


Running Procedures after migrating to Satellite 6.10
================================================================================
Start applicable services: 

Starting the following service(s):
rh-mongodb34-mongod, postgresql, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
- starting squid                                                                
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.
\ All services started                                                [OK]      
--------------------------------------------------------------------------------
re-enable sync plans: 
| Total 0 sync plans are now enabled.                                 [OK]      
--------------------------------------------------------------------------------
Remove maintenance_mode chain from iptables:                          [OK]
--------------------------------------------------------------------------------
SNIP


So i didn't see a failure, but the pulp2 services were started.  running foreman-maintain restart did not start them, so i think you're right amit, we need to 're-running' the feature detection after the installer runs?

Comment 13 Justin Sherrill 2021-06-23 21:53:43 UTC
Also to clarify, even though i didn't get an error, pulp2 services were spewing errors in the logs, so this still needs to be handled.

Comment 14 Justin Sherrill 2021-06-23 22:01:50 UTC
Connecting redmine issue https://projects.theforeman.org/issues/32870 from this bug

Comment 15 Devendra Singh 2021-06-24 05:09:02 UTC
(In reply to Justin Sherrill from comment #11)
> Devendra, did you see amit's question:  
> 
> 
> Devendra can you please confirm if you run : foreman-maintain service
> restart explicitly after the upgrade do you see pulp2 services getting
> restarted? and its specifically happening in the upgrade run's post actions?

Sorry I missed, yes after the upgrade all the satellite services were running, I restarted the services via foreman-maintain and it worked but not sure about pulp2 or pulp3.

Comment 17 Bryan Kearney 2021-06-29 20:05:56 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/32870 has been resolved.

Comment 18 Devendra Singh 2021-07-16 14:13:13 UTC
Verified on 6.10 Snap9

Comment 21 errata-xmlrpc 2021-11-16 13:48:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.10 Satellite Maintenance Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4697

Comment 22 Red Hat Bugzilla 2023-09-15 01:08:57 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.