Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1982686

Summary: Unable to upgrade with external mongodb
Product: Red Hat Satellite Reporter: Kenny Tordeurs <ktordeur>
Component: Satellite MaintainAssignee: Anurag Patel <apatel>
Status: CLOSED DUPLICATE QA Contact: Gaurav Talreja <gtalreja>
Severity: high Docs Contact:
Priority: high    
Version: 6.9.0CC: ahumbe, apatel, aupadhye, kgaikwad
Target Milestone: 6.10.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-19 16:46:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenny Tordeurs 2021-07-15 12:28:44 UTC
Description of problem:
foreman-maintain fails to work correctly with external Mongodb

Version-Release number of selected component (if applicable):
rubygem-foreman_maintain-0.7.10-1.el7sat.noarch
6.9.2.1 > 6.9.3

How reproducible:
100%

Steps to Reproduce:
1. External mongodb
2. Attempt upgrade (installs latest foreman-maintain)
3. 

Actual results:

~~~
# foreman-maintain service status
Running Status Services
================================================================================
Get status of applicable services: 

Displaying the following service(s):                                  [FAIL]
undefined method `enabled?' for RemoteDB(rh-mongodb34-mongod [5]):ForemanMaintain::Utils::Service::RemoteDB
--------------------------------------------------------------------------------
Scenario [Status Services] failed.

The following steps ended up in failing state:

  [service-status]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="service-status"
~~~

~~~
E, [2021-07-15 14:10:33+0200 #29315] ERROR -- : undefined method `enabled?' for RemoteDB(rh-mongodb34-mongod [5]):ForemanMaintain::Utils::Service::RemoteDB (NoMethodError)
/usr/share/gems/gems/foreman_maintain-0.7.10/definitions/features/service.rb:140:in `select!'
/usr/share/gems/gems/foreman_maintain-0.7.10/definitions/features/service.rb:140:in `filter_disabled_services!'
/usr/share/gems/gems/foreman_maintain-0.7.10/definitions/features/service.rb:133:in `filter_services'
/usr/share/gems/gems/foreman_maintain-0.7.10/definitions/features/service.rb:30:in `filtered_services'
/usr/share/gems/gems/foreman_maintain-0.7.10/definitions/procedures/service/base.rb:19:in `run_service_action'
/usr/share/gems/gems/foreman_maintain-0.7.10/definitions/procedures/service/start.rb:11:in `run'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/executable.rb:139:in `__run__'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner/execution.rb:83:in `block (2 levels) in run'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner/execution.rb:107:in `capture_errors'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner/execution.rb:82:in `block in run'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner/execution.rb:101:in `with_metadata_calculation'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner/execution.rb:81:in `run'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner.rb:131:in `run_step'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner.rb:120:in `run_steps'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner.rb:113:in `execute_scenario_steps'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/runner.rb:58:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:129:in `block in run_phase'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:177:in `with_non_empty_scenario'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:124:in `run_phase'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:164:in `block in rollback_pre_migrations'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:159:in `each'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:159:in `rollback_pre_migrations'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:90:in `run_rollback'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:66:in `block in run'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:65:in `each'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/upgrade_runner.rb:65:in `run'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/cli/upgrade_command.rb:96:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/foreman_maintain-0.7.10/lib/foreman_maintain/cli.rb:42:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
/usr/share/gems/gems/foreman_maintain-0.7.10/bin/foreman-maintain:15:in `<top (required)>'
/usr/bin/foreman-maintain:23:in `load'
/usr/bin/foreman-maintain:23:in `<main>'
~~~

Expected results:
No errors

Additional info:
downgrade to previous version of foreman-maintain fixes this:

Please disregard the older ktordeur-sat65 name it's running 6.9.2.1

# rpm -qa | grep maintain && foreman-maintain service status
satellite-maintain-0.0.1-1.el7sat.noarch
rubygem-foreman_maintain-0.7.8-1.el7sat.noarch
Running Status Services
================================================================================
Get status of applicable services: 

Displaying the following service(s):
rh-mongodb34-mongod, postgresql (candlepin), postgresql (foreman), postgresql, qdrouterd, qpidd, rh-redis5-redis, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy
/ displaying rh-mongodb34-mongod                                                
rh-mongodb34-mongod is remote and is UP.
- displaying postgresql (candlepin)                                             
postgresql (candlepin) is remote and is UP.
- displaying postgresql (foreman)                                               
postgresql (foreman) is remote and is UP.
- displaying postgresql                                                         
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/etc/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-07-04 15:51:36 CEST; 1 weeks 3 days ago
  Process: 26703 ExecStartPre=/opt/rh/rh-postgresql12/root/usr/libexec/postgresql-check-db-dir %N (code=exited, status=0/SUCCESS)
 Main PID: 26718 (postmaster)
    Tasks: 8
   Memory: 51.9M
   CGroup: /system.slice/postgresql.service
           ├─26718 postmaster -D /var/opt/rh/rh-postgresql12/lib/pgsql/data
           ├─26731 postgres: logger                                        
           ├─26733 postgres: checkpointer                                  
           ├─26734 postgres: background writer                             
           ├─26735 postgres: walwriter                                     
           ├─26736 postgres: autovacuum launcher                           
           ├─26737 postgres: stats collector                               
           └─26738 postgres: logical replication launcher                  

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
- displaying qdrouterd                                                          
● qdrouterd.service - Qpid Dispatch router daemon
   Loaded: loaded (/usr/lib/systemd/system/qdrouterd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/qdrouterd.service.d
           └─90-limits.conf
   Active: active (running) since Thu 2021-07-15 14:09:01 CEST; 13min ago
 Main PID: 1145 (qdrouterd)
    Tasks: 9
   Memory: 29.3M
   CGroup: /system.slice/qdrouterd.service
           └─1145 /usr/sbin/qdrouterd -c /etc/qpid-dispatch/qdrouterd.conf

Jul 15 14:09:06 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:06.346813 +0200 ROUTER_CORE (info) Link Route Activated 'linkRoute/1' on connection broker
Jul 15 14:09:06 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:06.346914 +0200 ROUTER_CORE (info) Link Route Activated 'linkRoute/2' on connection broker
Jul 15 14:09:06 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:06.346952 +0200 ROUTER_CORE (info) Link Route Activated 'linkRoute/3' on connection broker
Jul 15 14:09:10 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:10.646658 +0200 SERVER (info) [6]: Accepted connection to :5647 from 192.168.140.177:50608
Jul 15 14:09:11 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:11.169188 +0200 SERVER (info) [9]: Accepted connection to :5647 from 192.168.140.74:34618
Jul 15 14:09:11 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:11.173619 +0200 SERVER (info) [8]: Accepted connection to :5647 from 192.168.140.111:44094
Jul 15 14:09:11 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:11.275793 +0200 SERVER (info) [7]: Accepted connection to :5647 from 192.168.140.53:47892
Jul 15 14:09:11 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:11.478439 +0200 SERVER (info) [10]: Accepted connection to :5647 from 192.168.140.47:57682
Jul 15 14:09:11 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:11.480779 +0200 SERVER (info) [11]: Accepted connection to :5647 from 192.168.140.30:38102
Jul 15 14:09:11 ktordeur-sat65.sysmgmt.lan qdrouterd[1145]: 2021-07-15 14:09:11.539564 +0200 SERVER (info) [12]: Accepted connection to :5647 from 192.168.140.125:42476
- displaying qpidd                                                              
● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/qpidd.service.d
           └─90-limits.conf, wait-for-port.conf
   Active: active (running) since Thu 2021-07-15 14:09:02 CEST; 13min ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
  Process: 1154 ExecStartPost=/bin/bash -c while ! ss --no-header --tcp --listening --numeric sport = :5671 | grep -q "^LISTEN.*:5671"; do sleep 1; done (code=exited, status=0/SUCCESS)
 Main PID: 1153 (qpidd)
    Tasks: 10
   Memory: 25.7M
   CGroup: /system.slice/qpidd.service
           └─1153 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf

Jul 15 14:09:01 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting An AMQP message broker daemon....
Jul 15 14:09:02 ktordeur-sat65.sysmgmt.lan systemd[1]: Started An AMQP message broker daemon..
- displaying rh-redis5-redis                                                    
● rh-redis5-redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/rh-redis5-redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/rh-redis5-redis.service.d
           └─limit.conf
   Active: active (running) since Thu 2021-07-15 14:09:01 CEST; 13min ago
  Process: 866 ExecStop=/opt/rh/rh-redis5/root/usr/libexec/redis-shutdown (code=exited, status=0/SUCCESS)
 Main PID: 1147 (redis-server)
    Tasks: 4
   Memory: 7.2M
   CGroup: /system.slice/rh-redis5-redis.service
           └─1147 /opt/rh/rh-redis5/root/usr/bin/redis-server 127.0.0.1:6379                                       

Jul 15 14:09:01 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Redis persistent key-value database...
Jul 15 14:09:01 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Redis persistent key-value database.
- displaying squid                                                              
● squid.service - Squid caching proxy
   Loaded: loaded (/usr/lib/systemd/system/squid.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:09:01 CEST; 13min ago
  Process: 864 ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 1155 ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF (code=exited, status=0/SUCCESS)
  Process: 1146 ExecStartPre=/usr/libexec/squid/cache_swap.sh (code=exited, status=0/SUCCESS)
 Main PID: 1164 (squid)
    Tasks: 3
   Memory: 15.5M
   CGroup: /system.slice/squid.service
           ├─1164 /usr/sbin/squid -f /etc/squid/squid.conf
           ├─1166 (squid-1) -f /etc/squid/squid.conf
           └─1167 (logfile-daemon) /var/log/squid/access.log

Jul 15 14:09:01 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Squid caching proxy...
Jul 15 14:09:01 ktordeur-sat65.sysmgmt.lan squid[1164]: Squid Parent: will start 1 kids
Jul 15 14:09:01 ktordeur-sat65.sysmgmt.lan squid[1164]: Squid Parent: (squid-1) process 1166 started
Jul 15 14:09:01 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Squid caching proxy.
- displaying pulp_celerybeat                                                    
● pulp_celerybeat.service - Pulp's Celerybeat
   Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:09:02 CEST; 13min ago
 Main PID: 1222 (celery)
    Tasks: 5
   Memory: 50.5M
   CGroup: /system.slice/pulp_celerybeat.service
           └─1222 /usr/bin/python2 /usr/bin/celery beat --app=pulp.server.async.celery_instance.celery --scheduler=pulp.server.async.scheduler.Scheduler

Jul 15 14:09:02 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Pulp's Celerybeat.
Jul 15 14:09:03 ktordeur-sat65.sysmgmt.lan pulp[1222]: celery.beat:INFO: beat: Starting...
Jul 15 14:09:03 ktordeur-sat65.sysmgmt.lan pulp[1222]: pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
Jul 15 14:09:03 ktordeur-sat65.sysmgmt.lan pulp[1222]: pulp.server.db.connection:INFO: Attempting to connect to ktordeur-sat65-mongo.sysmgmt.lan:27017
Jul 15 14:09:03 ktordeur-sat65.sysmgmt.lan pulp[1222]: pulp.server.db.connection:INFO: Attempting to connect to ktordeur-sat65-mongo.sysmgmt.lan:27017
Jul 15 14:09:03 ktordeur-sat65.sysmgmt.lan pulp[1222]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
Jul 15 14:09:03 ktordeur-sat65.sysmgmt.lan pulp[1222]: pulp.server.async.worker_watcher:INFO: New worker 'scheduler.lan' discovered
Jul 15 14:19:04 ktordeur-sat65.sysmgmt.lan pulp[1222]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 15 14:19:04 ktordeur-sat65.sysmgmt.lan pulp[1222]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
- displaying pulp_resource_manager                                              
● pulp_resource_manager.service - Pulp Resource Manager
   Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:09:02 CEST; 13min ago
 Main PID: 1217 (celery)
    Tasks: 7
   Memory: 82.8M
   CGroup: /system.slice/pulp_resource_manager.service
           ├─1217 /usr/bin/python2 /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid
           └─1373 /usr/bin/python2 /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid

Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan celery[1217]: -------------- [queues]
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan celery[1217]: .> resource_manager exchange=resource_manager(direct) key=resource_manager
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan celery[1217]: .> resource_manager.lan.dq2 exchange=C.dq2(direct) key=resource_manager.lan
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp[1373]: pulp.server.db.connection:INFO: Attempting to connect to ktordeur-sat65-mongo.sysmgmt.lan:27017
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp[1373]: pulp.server.db.connection:INFO: Attempting to connect to ktordeur-sat65-mongo.sysmgmt.lan:27017
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp[1217]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp[1217]: celery.worker.consumer.connection:INFO: Connected to qpid://localhost:5671//
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp[1217]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp[1217]: celery.apps.worker:INFO: resource_manager.lan ready.
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp[1373]: pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
- displaying pulp_streamer                                                      
● pulp_streamer.service - The Pulp lazy content loading streamer
   Loaded: loaded (/usr/lib/systemd/system/pulp_streamer.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:09:02 CEST; 13min ago
 Main PID: 1211 (pulp_streamer)
    Tasks: 3
   Memory: 66.8M
   CGroup: /system.slice/pulp_streamer.service
           └─1211 /usr/bin/python /usr/bin/pulp_streamer --nodaemon --syslog --prefix=pulp_streamer --pidfile= --python /usr/share/pulp/wsgi/streamer.tac

Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: pulp.plugins.loader.manager:INFO: Loaded plugin puppet_whole_repo_profiler for types: puppet_module
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: pulp.plugins.loader.manager:INFO: Loaded plugin yum_profiler for types: rpm,erratum,modulemd
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: pulp.plugins.loader.manager:INFO: Loaded plugin yum for types: rpm
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: pulp.plugins.loader.manager:INFO: Loaded plugin rhui for types: rpm
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: [-] Loading /usr/share/pulp/wsgi/streamer.tac...
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: [-] Loaded.
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: [-] twistd 16.4.1 (/usr/bin/python 2.7.5) starting up.
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: [-] reactor class: twisted.internet.epollreactor.EPollReactor.
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: [-] Site starting on 8751
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan pulp_streamer[1211]: [-] Starting factory <twisted.web.server.Site instance at 0x7f6207e03d40>
- displaying pulp_workers                                                       
● pulp_workers.service - Pulp Celery Workers
   Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled; vendor preset: disabled)
   Active: active (exited) since Thu 2021-07-15 14:09:02 CEST; 13min ago
  Process: 775 ExecStop=/usr/bin/python2 -m pulp.server.async.manage_workers stop (code=exited, status=0/SUCCESS)
  Process: 1207 ExecStart=/usr/bin/python2 -m pulp.server.async.manage_workers start (code=exited, status=0/SUCCESS)
 Main PID: 1207 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
   CGroup: /system.slice/pulp_workers.service

Jul 15 14:09:02 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Pulp Celery Workers...
Jul 15 14:09:02 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Pulp Celery Workers.
- displaying smart_proxy_dynflow_core                                           
● smart_proxy_dynflow_core.service - Foreman smart proxy dynflow core service
   Loaded: loaded (/usr/lib/systemd/system/smart_proxy_dynflow_core.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/smart_proxy_dynflow_core.service.d
           └─90-limits.conf
   Active: active (running) since Thu 2021-07-15 14:09:04 CEST; 13min ago
     Docs: https://github.com/theforeman/smart_proxy_dynflow
 Main PID: 1206 (ruby)
    Tasks: 8
   Memory: 52.0M
   CGroup: /system.slice/smart_proxy_dynflow_core.service
           └─1206 ruby /usr/bin/smart_proxy_dynflow_core --no-daemonize

Jul 15 14:09:02 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Foreman smart proxy dynflow core service...
Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Foreman smart proxy dynflow core service.
- displaying tomcat                                                             
● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:09:02 CEST; 13min ago
 Main PID: 1226 (java)
    Tasks: 76
   Memory: 1.3G
   CGroup: /system.slice/tomcat.service
           └─1226 /usr/lib/jvm/jre/bin/java -Xms1024m -Xmx4096m -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Jul 15 14:09:20 ktordeur-sat65.sysmgmt.lan server[1226]: Jul 15, 2021 2:09:20 PM com.google.inject.internal.ProxyFactory <init>
Jul 15 14:09:20 ktordeur-sat65.sysmgmt.lan server[1226]: WARNING: Method [public org.candlepin.model.Persisted org.candlepin.model.RulesCurator.create(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@4820a7d0]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 15 14:09:21 ktordeur-sat65.sysmgmt.lan server[1226]: Jul 15, 2021 2:09:21 PM com.google.inject.internal.ProxyFactory <init>
Jul 15 14:09:21 ktordeur-sat65.sysmgmt.lan server[1226]: WARNING: Method [public void org.candlepin.model.EntitlementCertificateCurator.delete(org.candlepin.model.Persisted)] is synthetic and is being intercepted by [com.google.inject.persist.jpa.JpaLocalTxnInterceptor@4820a7d0]. This could indicate a bug.  The method may be intercepted twice, or may not be intercepted at all.
Jul 15 14:09:34 ktordeur-sat65.sysmgmt.lan server[1226]: Jul 15, 2021 2:09:34 PM org.apache.catalina.startup.HostConfig deployDirectory
Jul 15 14:09:34 ktordeur-sat65.sysmgmt.lan server[1226]: INFO: Deployment of web application directory /var/lib/tomcat/webapps/candlepin has finished in 30,344 ms
Jul 15 14:09:34 ktordeur-sat65.sysmgmt.lan server[1226]: Jul 15, 2021 2:09:34 PM org.apache.coyote.AbstractProtocol start
Jul 15 14:09:34 ktordeur-sat65.sysmgmt.lan server[1226]: INFO: Starting ProtocolHandler ["http-bio-127.0.0.1-23443"]
Jul 15 14:09:34 ktordeur-sat65.sysmgmt.lan server[1226]: Jul 15, 2021 2:09:34 PM org.apache.catalina.startup.Catalina start
Jul 15 14:09:34 ktordeur-sat65.sysmgmt.lan server[1226]: INFO: Server startup in 30427 ms
- displaying dynflow-sidekiq@orchestrator                                       
● dynflow-sidekiq - Foreman jobs daemon - orchestrator on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:10:23 CEST; 11min ago
     Docs: https://theforeman.org
 Main PID: 1410 (sidekiq)
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq
           └─1410 sidekiq 5.2.7  [0 of 1 busy]                                                                                                                                                       

Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jul 15 14:09:06 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@orchestrator[1410]: 2021-07-15T12:09:06.483Z 1410 TID-28f3q INFO: GitLab reliable fetch activated!
Jul 15 14:09:06 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@orchestrator[1410]: 2021-07-15T12:09:06.484Z 1410 TID-dln06 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-1410", :url=>"redis://localhost:6379/0"}
Jul 15 14:09:18 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@orchestrator[1410]: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
Jul 15 14:09:18 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@orchestrator[1410]: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Jul 15 14:09:42 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@orchestrator[1410]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-7.0.1.6/app/models/concerns/distributor_version.rb:5: warning: already initialized constant Katello::Glue::Provider::DISTRIBUTOR_VERSION
Jul 15 14:09:42 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@orchestrator[1410]: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/models/katello/glue/provider.rb:3: warning: previous definition of DISTRIBUTOR_VERSION was here
Jul 15 14:09:54 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@orchestrator[1410]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_azure_rm-2.1.3/app/models/foreman_azure_rm/azure_rm_compute.rb:13: warning: circular argument reference - sdk
Jul 15 14:10:23 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Foreman jobs daemon - orchestrator on sidekiq.
- displaying foreman                                                            
● foreman.service - Foreman
   Loaded: loaded (/usr/lib/systemd/system/foreman.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/foreman.service.d
           └─installer.conf
   Active: active (running) since Thu 2021-07-15 14:10:22 CEST; 11min ago
     Docs: https://theforeman.org
 Main PID: 1409 (ruby)
   Status: "puma 4.3.6 cluster: 2/2 workers: 0/32 threads, 32 available, 0 backlog"
    Tasks: 38
   Memory: 1.3G
   CGroup: /system.slice/foreman.service
           ├─1409 puma 4.3.6 (tcp://127.0.0.1:3000) [foreman]                      
           ├─2110 puma: cluster worker 0: 1409 [foreman]                           
           └─2117 puma: cluster worker 1: 1409 [foreman]                           

Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] * Min threads: 0, max threads: 16
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] * Environment: production
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] * Process workers: 2
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] * Preloading application
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] * Activated tcp://127.0.0.1:3000
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] Use Ctrl-C to stop
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Foreman.
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] * systemd: notify ready
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] - Worker 0 (pid: 2110) booted, phase: 0
Jul 15 14:10:22 ktordeur-sat65.sysmgmt.lan foreman[1409]: [1409] - Worker 1 (pid: 2117) booted, phase: 0
- displaying httpd                                                              
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:09:05 CEST; 13min ago
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 537 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
  Process: 5286 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
 Main PID: 1414 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
    Tasks: 164
   Memory: 401.9M
   CGroup: /system.slice/httpd.service
           ├─1414 /usr/sbin/httpd -DFOREGROUND
           ├─1486 (wsgi:pulp)     -DFOREGROUND
           ├─1487 (wsgi:pulp)     -DFOREGROUND
           ├─1488 (wsgi:pulp)     -DFOREGROUND
           ├─1489 (wsgi:pulp-cont -DFOREGROUND
           ├─1490 (wsgi:pulp-cont -DFOREGROUND
           ├─1491 (wsgi:pulp-cont -DFOREGROUND
           ├─1492 (wsgi:pulp_forg -DFOREGROUND
           ├─1493 PassengerWatchdog
           ├─1496 PassengerHelperAgent
           ├─1506 PassengerLoggingAgent
           ├─1516 /usr/sbin/httpd -DFOREGROUND
           ├─1517 /usr/sbin/httpd -DFOREGROUND
           ├─1518 /usr/sbin/httpd -DFOREGROUND
           ├─1519 /usr/sbin/httpd -DFOREGROUND
           ├─1522 /usr/sbin/httpd -DFOREGROUND
           ├─1523 /usr/sbin/httpd -DFOREGROUND
           ├─1524 /usr/sbin/httpd -DFOREGROUND
           ├─1525 /usr/sbin/httpd -DFOREGROUND
           └─2213 /usr/sbin/httpd -DFOREGROUND

Jul 15 14:09:07 ktordeur-sat65.sysmgmt.lan pulp[1486]: pulp.server.agent.direct.services:INFO: Task reply handler, started.
Jul 15 14:09:07 ktordeur-sat65.sysmgmt.lan pulp[1486]: pulp.server.agent.direct.services:INFO: AMQP reply handler started
Jul 15 14:09:07 ktordeur-sat65.sysmgmt.lan pulp[1486]: pulp.server.webservices.application:INFO: *************************************************************
Jul 15 14:09:07 ktordeur-sat65.sysmgmt.lan pulp[1486]: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized
Jul 15 14:09:07 ktordeur-sat65.sysmgmt.lan pulp[1486]: pulp.server.webservices.application:INFO: *************************************************************
Jul 15 14:09:08 ktordeur-sat65.sysmgmt.lan pulp[1486]: gofer.messaging.adapter.qpid.connection:INFO: opened: qpid+ssl://localhost:5671
Jul 15 14:09:08 ktordeur-sat65.sysmgmt.lan pulp[1486]: gofer.messaging.adapter.connect:INFO: connected: qpid+ssl://localhost:5671
Jul 15 14:15:22 ktordeur-sat65.sysmgmt.lan pulp[1487]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 15 14:17:46 ktordeur-sat65.sysmgmt.lan pulp[1487]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jul 15 14:17:46 ktordeur-sat65.sysmgmt.lan pulp[1486]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
- displaying puppetserver                                                       
● puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:09:41 CEST; 12min ago
  Process: 536 ExecStop=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver stop (code=exited, status=0/SUCCESS)
  Process: 1421 ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start (code=exited, status=0/SUCCESS)
 Main PID: 1537 (java)
    Tasks: 57 (limit: 4915)
   Memory: 1.2G
   CGroup: /system.slice/puppetserver.service
           └─1537 /usr/bin/java -Xms2G -Xmx2G -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger -XX:ReservedCodeCacheSize=512m -XX:OnOutOfMemoryError="kill -9 %p" -XX:ErrorFile=/var/log/puppetlabs/puppetserver/puppetserver_err_pid%p.log -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/services.d/,/opt/puppetlabs/server/apps/puppetserver/config/services.d/ --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter

Jul 15 14:09:04 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting puppetserver Service...
Jul 15 14:09:41 ktordeur-sat65.sysmgmt.lan systemd[1]: Started puppetserver Service.
- displaying dynflow-sidekiq@worker                                             
● dynflow-sidekiq - Foreman jobs daemon - worker on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:11:31 CEST; 10min ago
     Docs: https://theforeman.org
 Main PID: 2164 (sidekiq)
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq
           └─2164 sidekiq 5.2.7  [0 of 5 busy]                                                                                                                                                 

Jul 15 14:10:23 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Foreman jobs daemon - worker on sidekiq...
Jul 15 14:10:25 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker[2164]: 2021-07-15T12:10:25.509Z 2164 TID-2kfik INFO: GitLab reliable fetch activated!
Jul 15 14:10:25 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker[2164]: 2021-07-15T12:10:25.510Z 2164 TID-dynu4 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-2164", :url=>"redis://localhost:6379/0"}
Jul 15 14:10:36 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker[2164]: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
Jul 15 14:10:36 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker[2164]: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Jul 15 14:10:54 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker[2164]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-7.0.1.6/app/models/concerns/distributor_version.rb:5: warning: already initialized constant Katello::Glue::Provider::DISTRIBUTOR_VERSION
Jul 15 14:10:54 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker[2164]: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/models/katello/glue/provider.rb:3: warning: previous definition of DISTRIBUTOR_VERSION was here
Jul 15 14:11:04 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker[2164]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_azure_rm-2.1.3/app/models/foreman_azure_rm/azure_rm_compute.rb:13: warning: circular argument reference - sdk
Jul 15 14:11:31 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Foreman jobs daemon - worker on sidekiq.
- displaying dynflow-sidekiq@worker-hosts-queue                                 
● dynflow-sidekiq - Foreman jobs daemon - worker-hosts-queue on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:11:30 CEST; 10min ago
     Docs: https://theforeman.org
 Main PID: 2165 (sidekiq)
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq
           └─2165 sidekiq 5.2.7  [0 of 5 busy]                                                                                                                                                             

Jul 15 14:10:23 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Foreman jobs daemon - worker-hosts-queue on sidekiq...
Jul 15 14:10:25 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker-hosts-queue[2165]: 2021-07-15T12:10:25.507Z 2165 TID-5h305 INFO: GitLab reliable fetch activated!
Jul 15 14:10:25 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker-hosts-queue[2165]: 2021-07-15T12:10:25.511Z 2165 TID-gu2xp INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-2165", :url=>"redis://localhost:6379/0"}
Jul 15 14:10:36 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker-hosts-queue[2165]: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
Jul 15 14:10:36 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker-hosts-queue[2165]: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Jul 15 14:10:53 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker-hosts-queue[2165]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-7.0.1.6/app/models/concerns/distributor_version.rb:5: warning: already initialized constant Katello::Glue::Provider::DISTRIBUTOR_VERSION
Jul 15 14:10:53 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker-hosts-queue[2165]: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.29/app/models/katello/glue/provider.rb:3: warning: previous definition of DISTRIBUTOR_VERSION was here
Jul 15 14:11:03 ktordeur-sat65.sysmgmt.lan dynflow-sidekiq@worker-hosts-queue[2165]: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_azure_rm-2.1.3/app/models/foreman_azure_rm/azure_rm_compute.rb:13: warning: circular argument reference - sdk
Jul 15 14:11:30 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Foreman jobs daemon - worker-hosts-queue on sidekiq.
\ displaying foreman-proxy                                                      
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-07-15 14:11:36 CEST; 10min ago
 Main PID: 2518 (ruby)
    Tasks: 9
   Memory: 49.6M
   CGroup: /system.slice/foreman-proxy.service
           └─2518 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Jul 15 14:11:31 ktordeur-sat65.sysmgmt.lan systemd[1]: Starting Foreman Proxy...
Jul 15 14:11:36 ktordeur-sat65.sysmgmt.lan systemd[1]: Started Foreman Proxy.
\ All services are running                                            [OK]      
--------------------------------------------------------------------------------
~~~

Comment 1 Amit Upadhye 2021-07-19 16:46:59 UTC

*** This bug has been marked as a duplicate of bug 1978764 ***